Feature #7304

mega.co.nz Support

Added by coalado over 1 year ago. Updated over 1 year ago.

Status:Closed Start date:01/20/2013
Priority:Urgent Due date:
Assignee:coalado % Done:

100%

Category:Hoster
Target version:-
Resolution:

History

Updated by coalado over 1 year ago

From https://mega.co.nz/#developers :

1 Preface
Welcome to the MEGA API, and thank you for your interest. Whether you are just curious about the inner workings of MEGA or actually plan to write a client application, we hope that you will find it informative and comprehensible. This documentation, as well as the API itself, will evolve over time. If you have feedback or suggestions, we'd love to hear from you. Please write to developers@mega.co.nz.
In the near future, we will provide reference client libraries/SDKs for various programming languages. For now, our JavaScript site code is the only sample code available, and we did not have the time to comment, structure or even format it properly yet (it is also our first JavaScript project, so please bear with us).
1.1 Security considerations
The cryptographic integrity of MEGA's user data is important to us. We can therefore not allow you to distribute or make available your client application without going through us. We will perform a code audit of your product and promote/distribute it on our site. You will also receive a share of the revenue that your application generates.
1.2 Communication protocol
The MEGA API is based on a simple HTTP/JSON request-response scheme. Requests are submitted as arrays of command objects and can be initiated by both the client and the server, effectively resulting in bidirectional RPC capability.
All cryptographic operations involving symmetric or private keys are performed on the client side, largely eliminating the privacy concerns inherent to the traditional cloud storage model, but burdening the client with additional complexity.
1.3 Storage model
MEGA's filesystem uses the standard hierarchical file/folder paradigm. Each file and folder node points to a parent folder node, with the exception of three parent-less root folder nodes per user account - one for his personal files, one inbox for secure unauthenticated file delivery, and one rubbish bin.
Each general filesystem node (files/folders) has an encrypted attributes object attached to it, which typically contains just the filename, but will soon be used to transport user-to-user messages to augment MEGA's secure online collaboration capabilities.
As the server side is unable to see file and folder names, all operations involving them (such as versioning and collision handling) are up to the client. Referencing files or folders by their names requires reading and decrypting all candidate nodes until the desired ones are found.
The filesystem semantics currently supported are the equivalents of these POSIX calls:
open(O_RDONLY)
open(O_WRONLY|O_CREAT) with final file size specification
pread()
pwrite() with boundary restrictions and towards the file size specified at open()
close()
mkdir()
rename()
unlink()
symlink()
chdir()
readdir()
Support for non-O_CREAT open(O_WRONLY) is planned, using encrypted forking delta files.
1.4 Cryptography
All symmetric cryptographic operations are based on AES-128. It operates in cipher block chaining mode for the file and folder attribute blocks and in counter mode for the actual file data. Each file and each folder node uses its own randomly generated 128 bit key. File nodes use the same key for the attribute block and the file data, plus a 64 bit random counter start value and a 64 bit meta MAC to verify the file's integrity.
Each user account uses a symmetric master key to ECB-encrypt all keys of the nodes it keeps in its own trees. This master key is stored on MEGA's servers, encrypted with a hash derived from the user's login password.
File integrity is verified using chunked CBC-MAC. Chunk sizes start at 128 KB and increase to 1 MB, which is a reasonable balance between space required to store the chunk MACs and the average overhead for integrity-checking partial reads.
In addition to the symmetric key, each user account has a 2048 bit RSA key pair to securely receive data. Its private component is stored encrypted with the user's symmetric master key.
MEGA supports secure cross-account access to folders. The owner of the folder is solely responsible for managing access; shares are non-transitive. All participants in a shared folder gain cryptographic access through a common share-specific key, which is passed from the owner (theoretically, from anyone participating in the share, but this would create a significant security risk in the event of a compromise of the core infrastructure) to new participants through RSA. All keys of the nodes in a shared folder, including its root node, are encrypted to this share key. The party adding a new node to a shared folder is responsible for supplying the appropriate node/share-specific key.
MEGA supports secure unauthenticated data delivery. Any fully registered user can receive files or folders in their inbox through their RSA public key.
1.4 Authentication
The MEGA API supports two authentication modes:
1.4.1 Filesystem node
A node token ("magic cookie") grants access to a subtree of the issuing user's filesystem. An associated symmetric key is required to decrypt and/or store decryptable data. Procedurally, node token access is largely equivalent to shared folder operation between users and supports the same three permission levels. All filesystem access under this authentication mode is confined to the subtree underneath the token node.
1.4.2 User session
User session access grants full access to the user account and enables the full spectrum of API features.
1.5 User/session management
Accounts typically undergo three stages of completion:
Ephemeral
MEGA supports ephemeral accounts to enable pre-registration file manager operations, allowing applications to present MEGA's functionality with a lowered barrier to entry. They naturally do not have an e-mail address associated with them, the password (master key) is generated randomly, and they are subject to frequent purging. In a browser context, the credentials for ephemeral accounts live in the DOM storage.
Confirming
An ephemeral account can be upgraded to a confirming account by associating an e-mail address with it. A confirmation code is then e-mailed to the supplied address.
Complete
A confirming account is upgraded to a full account after the e-mailed confirmation code has been entered. The process includes the creation of an RSA key pair.
1.6 Sessions
Each login starts a new session. For complete accounts, this involves the server generating a random session token and encrypting it to the user's private key. The user password is considered verified if it successfully decrypts the private key, which then successfully decrypts the session token.
To prevent remote offline dictionary attacks on a user's password, the encrypted private key is only supplied to the client if a hash derived from the password is presented to the server.
2 API requests
API requests flow in two directions: client ? server and server ? client.
2.1 Client-server requests
Client-server requests are issued as HTTP POST with a raw JSON payload. A request consists of one or multiple commands and is executed as a single atomic, isolated and consistent transaction - in the event of a request-level error response, no data was altered. Requests are idempotent - sending the same request multiple times is equivalent to sending it once, which makes it safe to retry them, e.g. in case of intermittent network issues. Each request shall therefore be tagged with a session-unique identifier (e.g., a sequence number) to prevent inadvertent cache hits caused by preceding identical requests.
While a request is executed, all users that may be affected by it are locked. This includes the requesting user and all users that are in a shared folder relationship and/or in the contact list.
A request may return the error code EAGAIN in the event of a failed locking attempt or a temporary server-side malfunction. The request is likely to complete when retried. Client applications must implement exponential backoff (with user-triggerable immediate retry) and should inform the user of a possible server or network issue if the EAGAIN condition persists or no response is received for more than a few seconds.
A successfully executed request returns an array of result objects, with each result appearing in the same array index location as the corresponding command.
2.1.1 Request URL format
Target URL: https://g.api.mega.co.nz/cs?id=sequence_number[&sid=sessionid|&n=node]
sequence_number is a session-unique number that is incremented per dispatched request (but not changed when requests are repeated in response to network issues or EAGAIN)
sessionid is the session ID of the user session authentication mode
node is the node token of the filesystem tree authentication mode
2.1.2 POST JSON request payload structure
The JSON object shall be sent as the payload of a raw POST request. No additional framing shall take place. The Content-Type HTTP header is not processed, but should be set to application/json.
Its structure is an array of commands: [cmd1,cmd2,...]
with cmd = { a : command type, [argument : value]* }.
2.1.2 POST JSON response payload structure
The response to the request is a raw JSON object of content-type application/json.
It is structured as single number (e.g. -3 for EAGAIN) in the case of a request-level error or as an array of per-command return objects: [res1,res2,...]
To prevent infrastructure overload, dynamic rate limiting is in effect. Before a request is executed, the total "weight" of the commands it contains is computed and checked against the current balance of the requesting IP address. If the total exceeds a defined threshold, the request is rejected as a whole and must be repeated with exponential backoff.
2.2 Server-client requests
As a server cannot reliably establish a connection to a client, server-client requests have to be polled by the latter through a blocking read loop.
2.2.1 Request URL format
Target URL: https://g.api.mega.co.nz/sc?id=sequence_reference[&sid=sessionid|&n=node][ssl=1]
sequence_reference tells the server which server-client request(s) to deliver next. It is initialized from the response to a filesystem tree fetch (f command).
sessionid is the session ID of the user session authentication mode
node is the node token of the filesystem tree authentication mode
ssl=1 forces an HTTPS URL for the returned wait_url (which is needed for most browsers, but not in an application context)
2.2.2 POST JSON request payload structure
None.
2.2.3 POST JSON respone payload structure
A request level error is received as a single number (e.g. -3 for EAGAIN) or a raw JSON object with content-type application/json. Its structure is as follows:
{ a : [req1,req2,...], [ sn : sequence_reference | w : wait_url ] }
reqn are server-client requests
sequence_reference updates the sequence reference that is used the invocation of the /cs request URL
wait_url requests that the client connects to this (potentially long) URL, which will block until new requests are ready for delivery, so once it disconnects (with an HTTP 200 OK response and a content-length of 0), the polling process shall loop back to fetching new requests, using the current sequence_reference.
2.3 JSON data encoding
As JSON is not binary clean, all non-ASCII data has to be encoded. For binary data, the MEGA API uses a variation of base64 with -_ instead of +/ and the trailing = stripped (where necessary, the actual payload length is heuristically inferred after decoding, e.g. by stripping trailing NULs).
UNICODE text has to be encoded as UTF-8.
2.4 API data types
The MEGA API uses the following major data types:
2.4.1 Node handles
Node handles are eight alphanumeric characters and case sensitive.
2.4.2 User handles
User handles are eleven base64-characters.
2.4.3 Encryption keys
Encryption keys are always base64-encoded. The following key types exist:
Symmetric AES keys (22 characters)
Folder node keys (22 characters)
File node keys (43 characters)
RSA public/private keys (2048 bit: 348/875 characters)
Node and file keys in a share context are transmitted in a compound per-share format: sharehandle:key/sharehandle:key/... - each key is encrypted to its corresponding share handle.
2.5 Error codes
Requests can fail for a multitude of reasons. At this time, the following error codes are defined:
General errors:
EINTERNAL (-1): An internal error has occurred. Please submit a bug report, detailing the exact circumstances in which this error occurred.
EARGS (-2): You have passed invalid arguments to this command.
EAGAIN (-3) (always at the request level): A temporary congestion or server malfunction prevented your request from being processed. No data was altered. Retry. Retries must be spaced with exponential backoff.
ERATELIMIT (-4): You have exceeded your command weight per time quota. Please wait a few seconds, then try again (this should never happen in sane real-life applications).
Upload errors:
EFAILED (-5): The upload failed. Please restart it from scratch.
ETOOMANY (-6): Too many concurrent IP addresses are accessing this upload target URL.
ERANGE (-7): The upload file packet is out of range or not starting and ending on a chunk boundary.
EEXPIRED (-8): The upload target URL you are trying to access has expired. Please request a fresh one.
Filesystem/Account-level errors:
ENOENT (-9): Object (typically, node or user) not found
ECIRCULAR (-10): Circular linkage attempted
EACCESS (-11): Access violation (e.g., trying to write to a read-only share)
EEXIST (-12): Trying to create an object that already exists
EINCOMPLETE (-13): Trying to access an incomplete resource
EKEY (-14): A decryption operation failed (never returned by the API)
ESID (-15): Invalid or expired user session, please relogin
EBLOCKED (-16): User blocked
EOVERQUOTA (-17): Request over quota
ETEMPUNAVAIL (-18): Resource temporarily not available, please try again later
3 Client-server request reference
3.1 Filesystem operations
3.1.1 Retrieve folder or user nodes - f
Returns the contents of the requested folder, or a full view of the requesting user's three filesystem trees, contact list, incoming shares and pending share key requests.
3.1.2 Add/copy nodes - p
Adds new nodes. Copies existing files and adds completed uploads to a user's filesystem.
3.1.3 Delete node - d
Deletes a node, including all of its subnodes.
3.1.4 Move node - m
Moves a node to a new parent node.
3.1.5 Set node attributes - a
Updates the encrypted node attributes object.
3.1.6 Create/delete public handle - l
Enables or disables the public handle for a node.
3.1.7 Create/modify/delete outgoing share - s
Controls the sharing status of a node.
3.1.8 Key handling - k
Responds to or requests the following types of key processing:
cr - Set or request share/node keys
sr - Set share keys (in response to a share key request)
nk - Set node keys
3.1.9 Request download URL - g
Requests a temporary download URL for a file node.
3.1.10 Request upload URL - u
Requests a file-specific upload URL.
3.2 Account operations
3.2.1 Add/update user - up
Adds a new user, upgrades an existing user or sets/modifies a user's credentials.
3.2.2 Get user - ug
Retrieves user details.
3.2.3 Retrieve user's public key - uk
Retrieves a user's RSA public key.
3.2.4 Add/update/delete contact - ur
Modifes the contact status of a given user.
3.2.5 Invite user - ui
Sends invitation e-mail to a user.
3.2.6 Send confirmation e-mail - uc
Triggers the confirmation link to be sent to the registering user.
3.2.7 Obtain user details by invitation code - uv
Retrieves user details based on the invitation code.
3.2.8 Verify e-mailed confirmation code - ud
Upgrades an account's status based on the the given confirmation code.
3.2.9 Login session challenge/response - us
Establishes a user session based on the response to a cryptographic challenge.
3.2.10 List user sessions - usl
Retrieves the user's session history.
3.2.11 User quota details - uq
Returns the current quota and resource utilization for the user and for the requesting IP address.
4 Server-client request reference
4.1 Node addition - t
New nodes have arrived and need to be added to the filesystem view.
4.2 Node deletion - d
Existing nodes have been deleted and need to be removed from the filesystem view.
4.3 Share addition/modification/deletion - s
A share was added, modified or deleted. The filesystem view and the user contact list need to be updated accordingly.
4.4 Crypto request - k
A client-side cryptographic operation was requested (e.g., a share invitee just created his RSA key pair and is now ready to receive the share secret).
5 File encryption and integrity checking
MEGA uses client-side encryption/decryption to end-to-end-protect file transfers and storage. Data received from clients is stored and transmitted verbatim; servers neither decrypt, nor re-encrypt, nor verify the encryption of incoming user files. All cryptographic processing is under the control of the end user.
To allow for integrity-checked partial reads, a file is treated as a series of chunks. To simplify server-side processing, partial uploads can only start and end on a chunk boundary. Furthermore, partial downloads can only be integrity-checked if they fulfil the same criterion.
Chunk boundaries are located at the following positions:
0 / 128K / 384K / 768K / 1280K / 1920K / 2688K / 3584K / 4608K / ... (every 1024 KB) / EOF
A file key is 256 bits long and consists of the following components:
A 128 bit AES-128 key k
The upper 64 bit n of the counter start value (the lower 64 bit are starting at 0 and incrementing by 1 for each AES block of 16 bytes)
A 64 bit meta-MAC m of all chunk MACs
A chunk MAC is computed as follows (this is essentially CBC-MAC, which was chosen instead of the more efficient OCB over intellectual property concerns):
h := (n << 64) + n
For each AES block d: h := AES(k,h XOR d)
A chunk is encrypted using standard counter mode:
For each AES block d at block position p: d' := d XOR AES(k,(n << 64)+p)
MAC computation and encryption can be performed in the same loop.
Decryption is analogous.
To obtain the meta-MAC m, apply the same CBC-MAC to the resulting block MACs with a start value of 0. The 64 bit meta-MAC m is computed as ((bits 0-31 XOR bits 32-63) << 64) + (bits 64-95 XOR bits 96-127).
5.1 Uploads
Uploads are performed by POSTing raw data to the target URL returned by the API u command. If so desired, an upload can be performed in chunks. Chunks can be sent in any order and can be of any size, but they must begin and end on a chunk boundary. The byte offset x of a chunk within the file is indicated by appending /x to the URL. Multiple chunks can be sent in parallel. After a chunk completes, the server responds with a status message, which can be
Empty - successful receipt
A (negative) error code in decimal ASCII representation, typically requiring a restart of the upload from scratch
A 27-character base64-encoded completion handle that must be used in conjunction with the p (put node) API command to complete the upload
The per-upload encryption key must be generated by a strong random number generator. Using a weak one will undermine the confidentiality and integrity of your data.
5.2 Downloads
Downloads are performed by GETting from the source URL returned by the API g command. Partial requests via the HTTP Range header are supported. Alternatively, the desired byte range can be appended to the source URL:
URL/1000 requests all data from offset 1000 to the end
URL/1000-2000 requests all data from offset 1000 up to and including offset 2000 (1001 bytes)
Multiple download connections can run in parallel.
Errors are indicated through the HTTP response code:
503 - too many connections for this download
509 - transfer quota exceeed
5.3 Parallel connections and TCP performance considerations
TCP throughput on long latency links is adversely affected by slow congestion window growth, insufficient send or receive buffer size and (even mild) packet loss. All of these factors can be mitigated by using multiple transfer connections in parallel. Client applications are encouraged to offer users to configure up to six parallel connections in each direction. The recommended default value is four.
5.4 HTTPS vs. HTTP
All MEGA servers support SSLv3/TLS access - this is due to many web browsers enforcing a policy where HTTP requests cannot be made from an HTTPS page at all (IE, Firefox 18+) or at least trigger a visual warning (Chrome, Firefox until 17). However, only two types of requests actually benefit from and therefore require HTTPS: The loading of https://mega.co.nz/index.html (plus three Web Worker-related .js files) and the API request interface. Neither the hash-protected loading of static .html and .js components, nor the waiting for new server-client requests, nor already encrypted and MAC'ed data transfers from and to the storage cluster benefit from HTTPS in any meaningful way. Client applications are therefore required to use SSL for access to the API interface, but strongly discouraged from doing so for wait requests and bulk file transfers.
MEGA HTTPS access uses RC4-MD5 (which minimizes CPU load without sacrificing security) with 2048 bit RSA where security is relevant and 1024 bit RSA where it is not.
6 Mandatory acceptance of the MEGA Terms of Service by application users
Client applications must present the user with the MEGA terms of service located at http://g.static.mega.co.nz/pages/terms.html and require their acceptance them before a newly created account (even if it is an ephemeral account) can be used.

Updated by pspzockerscene over 1 year ago

  • Priority changed from Normal to Urgent

Urgent priority, people want it!

Updated by pspzockerscene over 1 year ago

Should I try or will someone else?
I probably need too much time :(

Updated by coalado over 1 year ago

We figured out how to download.
A Plugin will be available soon

Updated by pspzockerscene over 1 year ago

  • Status changed from New to Assigned
  • Assignee set to coalado

Nice!
How do you like their crypting stuff? :D

Updated by jiaz over 1 year ago

  • Status changed from Assigned to Closed
  • % Done changed from 0 to 100

Applied in changeset r19533.

Also available in: Atom PDF