Metalink support

Hello ..

I've quickly wrote an importer for the metalink v3 format. To test it download the latest version from
https://www.smartftp.com/download
Then in the Transfer Queue tab, click on the Add button. Then select the "*.metalink" file. The import adds the files to the Temporary Queue because they don't have a destination yet.

The metalink format is not very well designed for the FTP protocol. For example it doesn't support FTP over SSL etc.

Regards,
SmartFTP

As I understand it, it does not make much sense to support Metalink in SmartFTP, because SmartFTP cannot (yet) download a single file from multiple servers, and that's the main purpose of Metalink: To list mirrors of a single file.

So it's important to understand SmartFTP will not parse all URLs listed in a <resources> section, but only add one of them (the first FTP URL). It only adds another URL if it occurs in a different <files> section. Put simply: Currently only adding one URL per <resources> section is supported.

Hello ...

>What do you need besides <url type="ftps">? Since metalinks are for listing the types that client programs support, it will be easy enough to add.
I think this should be sufficient. Right now SmartFTP will try to parse the URL if the type is set to "ftp" or "ftps". If the URL starts with "ftps://" SmartFTP assumes it's FTP over SSL.

>Just curious, I'm not familiar w/ XMD5 or XSHA which I see SmartFTP supports. I guess these are FTP server commands similar to Content-MD5 headers from web servers?
Yes that's correct. The client sends XMD5 <file> and the server returns the MD5 hash value. Unfortunately FTP servers under *nix do not support such features. However most FTP servers on MS Windows do.

>Will SmartFTP be able to use the MD5 or SHA1 checksums listed in metalinks to verify the downloads (or does it already)?
Yep the latest version (2.0.1001.26) does now ;-) Please give it a try.
https://www.smartftp.com/download

Regards,
-Mat

Thank you for the additional information on the partial checksums. However I think we will delay the support for partial checksums until we have the multi-part dowload feature ready.
I would appreciate if you could let me know about any future additions to the Metalink format which may benefit an FTP client like ours.
Best Regards,
-Mat

Thank you for the additional information on the partial checksums. However I think we will delay the support for partial checksums until we have the multi-part dowload feature ready.
I would appreciate if you could let me know about any future additions to the Metalink format which may benefit an FTP client like ours.

I will. If you see any other areas of Metalink that need improvement, feedback is good

That will be great if you add multi-part downloads! It can really speed up downloads quite a bit.

Thank you for the updates :-)
-Mat

A new, but slightly different and incompatible version of Metalink is now an IETF Proposed Standard: RFC 5854

Would you consider supporting the new format?

Changes from current Metalink 3.0 specification:

  • Different namespace: xmlns="urn:ietf:params:xml:ns:metalink" (Previous: xmlns="http://www.metalinker.org")
  • Different file extension: .meta4
  • Different date format: RFC 3339. Example: 2008-12-13T18:30:02Z
  • <metalink> attributes (published, origin, generator, updated) are now elements.
  • <url> no longer has type attribute listing ftp, http, bittorrent.
  • New <metaurl> instead of URLs to torrents or metalinks.
  • Piece checksums are no longer numbered.
  • Removed extraneous empty container elements: <files>, <verification>, <resources>
  • Deprecated elements (<changelog>, <tags>, <mimetype>, <relations>, <releasedate>, <mulitmedia>, <screenshot>, <upgrade>, <bittorrent>, others?) and attributes ("maxconnections").
  • IANA registry Hash Function Textual Names defines values for hash types. The SHA hashes now have a "-" in them: sha-1, sha-224 sha-256, sha-384, sha-512 instead of "sha1" etc.
  • IANA registry Operating System Names defines values for OS types.

Example Metalink RFC (.meta4 extension) file:
   <?xml version="1.0" encoding="UTF-8"?>

   <metalink xmlns="urn:ietf:params:xml:ns:metalink">

     <file name="example.ext">

       <size>14471447</size>

       <url>ftp://ftp.example.com/example.ext</url>

       <url>http://example.com/example.ext</url>

       <metaurl mediatype="torrent">

       http://example.com/example.ext.torrent</metaurl>

     </file>

   </metalink>

PS - You are mentioned in Appendix A

Hello Anthony ..

Thanks for the update and congratulations for the proposed standard :-) I have implemented meta4 support in the latest version of SmartFTP:
https://www.smartftp.com/download
It also supports sha-256, sha-384 and sha-512 hash values now.

I'm wondering what were the reasons to remove the collection tags (e.g. files, resources etc)? Simplicity?

Best
Mat

awesome, that was fast as usual!

thanks, it only took about 2 years to move thru the IETF...whew

yes, those collection tags were just removed for simplicity.