Jump to content


Photo

Metalink support


  • Please log in to reply
12 replies to this topic

#1 twanj

twanj
  • Members
  • 9 posts

Posted 19 January 2007 - 05:33 PM

I noticed this in the changelog:

Version 2.0 Build 999 (6. October 2006)
Major Changes

* Transfer Queue Enhancements (XML import/export, Auto Save, Move, Delete, Local to Local)

2.0.998.4 - 5. September 2006 (BETA)
# Rewrote "Transfer Queue" back-end. New features include "Remove finished", "Auto Save" options, XML serialization/import and selection of columns in the list control.


It would be cool if Metalink support was added. It's an XML file that lists multiple mirrors for redundancy or segmented downloads, along with multiple files for transfer queues. It also lists checksums and other useful info. Download managers such as GetRight, Speed Download, FlashGot, etc support it.

Here's what it looks like:
<metalink version="3.0" xmlns="http://www.metalinker.org/">
  <files>
	<file name="example.ext">
	<verification>
	  <hash type="md5">example-md5-hash</hash>
	  <hash type="sha1">example-sha1-hash</hash>
	  <signature type="pgp"/>
	</verification>
	<resources>
	  <url type="ftp" location="us" preference="90">ftp://ftp.example1.com/example.ext</url>
	  <url type="ftp" location="uk" preference="90">ftp://ftp.example2.com/example.ext</url>
	  <url type="http" location="us" preference="90">http://www.example1.com/example.ext</url> 
	  <url type="http" location="uk" preference="90">http://www.example2.com/example.ext</url>
	  <url type="http" location="de" preference="90">http://www.example3.com/example.ext</url> 
	</resources>
	</file>
  </files>
</metalink>

(For queues, there are multiple "<file name="example.ext">" entries).

More info is at http://en.wikipedia.org/wiki/Metalink

#2 mb

mb

    Developer

  • Administrators
  • 11527 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 19 January 2007 - 11:24 PM

Hello ..

I've quickly wrote an importer for the metalink v3 format. To test it download the latest version from
http://www.smartftp....get/SFTPNSI.exe
Then in the Transfer Queue tab, click on the Add button. Then select the "*.metalink" file. The import adds the files to the Temporary Queue because they don't have a destination yet.

The metalink format is not very well designed for the FTP protocol. For example it doesn't support FTP over SSL etc.

Regards,
SmartFTP

#3 eyebex

eyebex
  • Licensed User
  • 1860 posts
  • Gender:
    Male

Posted 19 January 2007 - 11:54 PM

As I understand it, it does not make much sense to support Metalink in SmartFTP, because SmartFTP cannot (yet) download a single file from multiple servers, and that's the main purpose of Metalink: To list mirrors of a single file.

So it's important to understand SmartFTP will not parse all URLs listed in a <resources> section, but only add one of them (the first FTP URL). It only adds another URL if it occurs in a different <files> section. Put simply: Currently only adding one URL per <resources> section is supported.

#4 twanj

twanj
  • Members
  • 9 posts

Posted 20 January 2007 - 01:59 AM

Hello ..

I've quickly wrote an importer for the metalink v3 format. To test it download the latest version from
http://www.smartftp....get/SFTPNSI.exe
Then in the Transfer Queue tab, click on the Add button. Then select the "*.metalink" file. The import adds the files to the Temporary Queue because they don't have a destination yet.

The metalink format is not very well designed for the FTP protocol. For example it doesn't support FTP over SSL etc.


Haha, WOW, that was pretty quick! Less than 6 hrs <_< I tried it out, very cool!

I don't think any of the other metalink clients support anything besides straight FTP, but I may be wrong.

Are you talking about <url type="????">? The type is not a requirement (altho most autogenerated metalinks will have it, which is most of them), clients just use <url> and the programs can tell from the URL whether it begins w/ ftp://, http://, magnet, or ends in .torrent what type it is.

What do you need besides <url type="ftps">? Since metalinks are for listing the types that client programs support, it will be easy enough to add.

Do you have any other suggestions?

As I understand it, it does not make much sense to support Metalink in SmartFTP, because SmartFTP cannot (yet) download a single file from multiple servers, and that's the main purpose of Metalink: To list mirrors of a single file.

So it's important to understand SmartFTP will not parse all URLs listed in a <resources> section, but only add one of them (the first FTP URL). It only adds another URL if it occurs in a different <files> section. Put simply: Currently only adding one URL per <resources> section is supported.


I understand.

Yes, most programs that support metalink are download managers that allow for segmented multi-source downloads. But, listing the mirrors can also be nice for redundancy - if a mirror goes down during transfer, clients switch to another automatically. Or if a mirror(s) is already down (or already has its limit of anonymous connections, etc) when the transfer starts, it can go through them until it finds a working mirror. That may not fit into your current architecture too easy, so it may not be worth it.

Just being able to use it for adding files to the transfer queue is pretty neat. Things like GNOME or KDE have a bunch of files you would normally download together, with this you can grab them in one link and email it around or whatever...

Just curious, I'm not familiar w/ XMD5 or XSHA which I see SmartFTP supports. I guess these are FTP server commands similar to Content-MD5 headers from web servers? Will SmartFTP be able to use the MD5 or SHA1 checksums listed in metalinks to verify the downloads (or does it already)?

#5 mb

mb

    Developer

  • Administrators
  • 11527 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 21 January 2007 - 12:26 AM

Hello ...

>What do you need besides <url type="ftps">? Since metalinks are for listing the types that client programs support, it will be easy enough to add.
I think this should be sufficient. Right now SmartFTP will try to parse the URL if the type is set to "ftp" or "ftps". If the URL starts with "ftps://" SmartFTP assumes it's FTP over SSL.

>Just curious, I'm not familiar w/ XMD5 or XSHA which I see SmartFTP supports. I guess these are FTP server commands similar to Content-MD5 headers from web servers?
Yes that's correct. The client sends XMD5 <file> and the server returns the MD5 hash value. Unfortunately FTP servers under *nix do not support such features. However most FTP servers on MS Windows do.

>Will SmartFTP be able to use the MD5 or SHA1 checksums listed in metalinks to verify the downloads (or does it already)?
Yep the latest version (2.0.1001.26) does now ;-) Please give it a try.
http://www.smartftp....get/SFTPNSI.exe

Regards,
-Mat

#6 twanj

twanj
  • Members
  • 9 posts

Posted 21 January 2007 - 07:28 PM

Hi Mat,

Hello ...

>What do you need besides <url type="ftps">? Since metalinks are for listing the types that client programs support, it will be easy enough to add.
I think this should be sufficient. Right now SmartFTP will try to parse the URL if the type is set to "ftp" or "ftps". If the URL starts with "ftps://" SmartFTP assumes it's FTP over SSL.


I'll explicitly list type="ftps" - if there are any other improvements or additions, please share!

>Just curious, I'm not familiar w/ XMD5 or XSHA which I see SmartFTP supports. I guess these are FTP server commands similar to Content-MD5 headers from web servers?
Yes that's correct. The client sends XMD5 <file> and the server returns the MD5 hash value. Unfortunately FTP servers under *nix do not support such features. However most FTP servers on MS Windows do.

I hadn't heard of it before, that's a shame tho. It seems pretty useful, you'd think it would be easy to add, especially in the *nix culture.

>Will SmartFTP be able to use the MD5 or SHA1 checksums listed in metalinks to verify the downloads (or does it already)?
Yep the latest version (2.0.1001.26) does now ;-) Please give it a try.
http://www.smartftp....get/SFTPNSI.exe


Talk about rapid response! I tried it out and it works! Very nice.

Here's another thing which may not fit into your checksum verification code but you may find interesting - chunk or partial file checksums (as opposed to full file checksums).

You can see it in this file http://www.metalinke...se.iso.metalink (The other clients are adding this feature now, the chunk checksums can be generated with Metalink tools).

<verification>
			<hash type="md5">640145695ac7506a8357a3abc2ca442e</hash>
			<hash type="sha1">7045ca2ff5d69a45fe9ae4eafcc2832338f05e2f</hash>
			<pieces length="1048576" type="sha1">
				<hash piece="0">e8230ecbc94375b8a4b8e3b8da7d376c09d14d8e</hash>
				<hash piece="1">37517b42ed7353eb4b0c59f901d33197778343d9</hash>
				<hash piece="2">0e3edbaae7207ee8fb1439304a09c6cd95d01c57</hash>
								etc

As you can imagine, not as useful for a 5 MB file or so, but really nice for large stuff like ISOs. As the file downloads, each chunk or segment (in this example, a meg, but it varies due to total file size) can be verified. If it's bad, retransmit that byte range. If its bad more than once, use another mirror (I know SmartFTP only uses one URL, just an example).

With that, you can guarantee error free downloads for people who use it, and lower bandwidth costs for sites since a whole file does not need to be re-downloaded (double the bandwidth and download time) when theres an error, just a small portion is re-downloaded, and it's automatic so the user doesn't even need to do anything or need to know whats going on.

#7 mb

mb

    Developer

  • Administrators
  • 11527 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 22 January 2007 - 06:25 PM

Thank you for the additional information on the partial checksums. However I think we will delay the support for partial checksums until we have the multi-part dowload feature ready.
I would appreciate if you could let me know about any future additions to the Metalink format which may benefit an FTP client like ours.
Best Regards,
-Mat

#8 twanj

twanj
  • Members
  • 9 posts

Posted 06 February 2007 - 06:06 PM

Thank you for the additional information on the partial checksums. However I think we will delay the support for partial checksums until we have the multi-part dowload feature ready.
I would appreciate if you could let me know about any future additions to the Metalink format which may benefit an FTP client like ours.


I will. If you see any other areas of Metalink that need improvement, feedback is good <_<

That will be great if you add multi-part downloads! It can really speed up downloads quite a bit.

#9 twanj

twanj
  • Members
  • 9 posts

Posted 27 February 2007 - 06:19 PM

Here are some recent posts on metalink. The first one is particularly good.

http://www.geospatia...ernet-downloads

http://slashdot.org/...07/02/25/144209

#10 mb

mb

    Developer

  • Administrators
  • 11527 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 27 February 2007 - 09:08 PM

Thank you for the updates :-)
-Mat

#11 twanj

twanj
  • Members
  • 9 posts

Posted 07 June 2010 - 11:31 PM

A new, but slightly different and incompatible version of Metalink is now an IETF Proposed Standard: RFC 5854

Would you consider supporting the new format?

Changes from current Metalink 3.0 specification:

  • Different namespace: xmlns="urn:ietf:params:xml:ns:metalink" (Previous: xmlns="http://www.metalinker.org")
  • Different file extension: .meta4
  • Different date format: RFC 3339. Example: 2008-12-13T18:30:02Z
  • <metalink> attributes (published, origin, generator, updated) are now elements.
  • <url> no longer has type attribute listing ftp, http, bittorrent.
  • New <metaurl> instead of URLs to torrents or metalinks.
  • Piece checksums are no longer numbered.
  • Removed extraneous empty container elements: <files>, <verification>, <resources>
  • Deprecated elements (<changelog>, <tags>, <mimetype>, <relations>, <releasedate>, <mulitmedia>, <screenshot>, <upgrade>, <bittorrent>, others?) and attributes ("maxconnections").
  • IANA registry Hash Function Textual Names defines values for hash types. The SHA hashes now have a "-" in them: sha-1, sha-224 sha-256, sha-384, sha-512 instead of "sha1" etc.
  • IANA registry Operating System Names defines values for OS types.

Example Metalink RFC (.meta4 extension) file:
<?xml version="1.0" encoding="UTF-8"?>
   <metalink xmlns="urn:ietf:params:xml:ns:metalink">
     <file name="example.ext">
       <size>14471447</size>
       <url>ftp://ftp.example.com/example.ext</url>
       <url>http://example.com/example.ext</url>
       <metaurl mediatype="torrent">
       http://example.com/example.ext.torrent</metaurl>
     </file>
   </metalink>

PS - You are mentioned in Appendix A :D

#12 mb

mb

    Developer

  • Administrators
  • 11527 posts
  • Gender:
    Male
  • Location:
    Worldwide

Posted 08 June 2010 - 01:05 PM

Hello Anthony ..

Thanks for the update and congratulations for the proposed standard :-) I have implemented meta4 support in the latest version of SmartFTP:
http://www.smartftp.com/download
It also supports sha-256, sha-384 and sha-512 hash values now.

I'm wondering what were the reasons to remove the collection tags (e.g. files, resources etc)? Simplicity?

Best
Mat

#13 twanj

twanj
  • Members
  • 9 posts

Posted 08 June 2010 - 07:20 PM

awesome, that was fast as usual! :D

thanks, it only took about 2 years to move thru the IETF...whew :)

yes, those collection tags were just removed for simplicity.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users