macOS URL Loading System to Request Contents of a Directory?

[Spike099]

So, a little history about why this has stricken my curiosity.

I had a client who wanted to transfer all his geocity files over to a real provider. The client had thousands of files on his account. Now, geocities doesn't allow you to do bulk transfers, nor can you access the site via ftp on a basic account. So... Basically the only way for a typical person to do this was to download each file individually! This wasn't going to happen.

My resolution was to write an app that took in a csv list and downloaded each file for you.

This worked, however I had to copy and paste the list of files and format it properly.

My question is, is there a way to request the contents of a directory via URL Loading System or some other Cocoa or foundation framework?

 

[kainjow]

If you're referring to a directory on the server, no, not unless you go through some protocol like FTP or WebDAV.

 

[Spike099]

Yes, I was referring to a directory on the server.

So, a crawler would start by parsing whatever index (if no robot.txt) was provided and search the file for links, is this how they find all the pages available on a webserver?

 

[xyzeugene]

Dont reinvent the wheel...

Use http://www.httrack.com

its free and you and slow it down if need be.

So, a little history about why this has stricken my curiosity.

I had a client who wanted to transfer all his geocity files over to a real provider. The client had thousands of files on his account. Now, geocities doesn't allow you to do bulk transfers, nor can you access the site via ftp on a basic account. So... Basically the only way for a typical person to do this was to download each file individually! This wasn't going to happen.

My resolution was to write an app that took in a csv list and downloaded each file for you.

This worked, however I had to copy and paste the list of files and format it properly.

My question is, is there a way to request the contents of a directory via URL Loading System or some other Cocoa or foundation framework?

 

[Consultant]

Use http://www.httrack.com

its free and you and slow it down if need be.

Not sure about that one but there are plenty of apps that will download all linked files from a server.