Building with Sense API

Search…
This document describes how Sense can be used and configured across various external backends.
Background
Any existing platform, project, or protocol can access the NFT fingerprints and their relative rareness scores, and add such  to a registry of NFTs as a way to timestamp or “claim” the underlying data in a decentralized, trust-less way - without needing to migrate their existing backend infrastructure.
The user first creates a PastelID for their underlying project, which will be used to submit all External_Sense_Request API calls on the Pastel Network. Each call submits a request via gRPC from Pastel's WalletNode software that runs as part of the client to the network of Pastel SuperNodes, which then return the Relative Rareness Score in a JSON.
Each time the user calls External_Sense_Request the NFT is passed and added to a dedicated SQLite database hosted by the SuperNodes on Pastel. The JSON results returned from the API request are compared among the multiple SuperNodes randomly assigned to perform such calculations. If each result matches, the top-ranked SuperNode adds the JSON file to Pastel's Storage Layer via Kademlia. We then create a new Pastel Blockchain ticket, the External_Sense_Request_Ticket, which is signed by both the user's PastelID and the PastelIDs of each SuperNode that handled the request. This Pastel Blockchain ticket  contains the SHA3-256 hash of the JSON file in Kademlia containing the results of running the Sense Protocol on the NFT as well as the SHA3-256 hash of the original NFT data. The same NFT hash is included in the JSON metadata of the corresponding NFT on the user's platform.
Once the Sense Protocol is completed by the Supernodes, they create and sign the External_Sense_Request_Ticket with their PastelIDs.The user then signs the External_Dupe_Detection_Registration_Ticket with the user's PastelID, at which point the final signed External_Dupe_Detection_Registration_Ticket would be written to the Pastel blockchain by the Supernodes which counter-signed it. 

At this point, the user has an External_Dupe_Detection_Registration_Ticket on Pastel. The NFT provided by the user also has its own fingerprint vector added to the SQLite database. Thus, the next time that relatively same image is submitted to Pastel in a new External_Dupe_Detection_Request, the new NFT would result in a lower relative rareness score as it would be compared to the previous ticket for the similar NFT. In addition, the other data resulting from running the Sense Protocol on the image, such as the rareness score and the NFT perceptual hashes, would all be stored in Kademlia in a JSON file. To ensure redundancy, we create several variants of this JSON  by adding a nonce to the end of each one, resulting in a different SHA3-256 hash for each copy, which are then all stored in Kademlia separately. 

Because only Supernodes in Pastel can directly access Kademlia, we could set up some public-facing servers that are also Pastel Supernodes with a REST API that would respond to public requests; in these requests, users could submit an image file hash and get as a result the corresponding JSON file from Kademlia, assuming that this particular image had been submitted previously. But to avoid needing to rely on some external service, Rarible might elect to set up their own Pastel Supernode so that they could set up an API like that on their own and cover the cost of serving it at scale.
Summary:
External_Sense_Request (made via gRPC):
The PastelID of the user submitting the request.
1.PastelID of the user submitting the request
2.SHA3-256 hash of the candidate NFT file 
3.The External_Dupe_Detection_Sending_Address.
4.The Pastel signature of the user on the above data.
5.The Pastel signatures of the 3 top-ranked Supernodes at the time the request is submitted on the above data.
6.The SHA3-256 hash of all of the above data, which would serve as a unique identifier for the request, the Request_Identifier.
​
1.External_Dupe_Detection_Registration_Ticket (made via a Pastel Blockchain Ticket):
1.The Request_Identifier of the corresponding External_Dupe_Detection_Request.
2.The SHA3-256 hash of the image file the user wants to run dupe-detection on.
3.The Maximum_PSL_Fee_for_External_Dupe_Detection.
4.The External_Dupe_Detection_Sending_Address.
5.The effective total fee paid by the user in PSL (both paid to Supernodes and burned).
6.The SHA3-256 hash for the JSON file stored in Kademlia which itself contains the SHA3-256 hashes of the set of redundant copies of the JSON files in Kademlia (which contain the results of the dupe detection computation).
7.The Pastel signatures of the Supernodes that performed the dupe detection computations for the ticket on the above data.
8.The Pastel signature of the user on the above data.
9.The Pastel TXID of this transaction would then serve as the identifier for this ticket.
​


External Storage Layer API:A very similar scheme can be implemented for exposing the Pastel storage layer. Basically, you would have an External_Storage_Request (also submitted by the user via gRPC to the Supernode network) that would specify the SHA-256 hash of the file the user wanted to store (let’s again use the example of Rarible as the user in this case) as well as the size of this file in megabytes; this ticket would include a field for Maximum_PSL_Fee_per_Megabyte_for_Storage.
One basic question is whether we want to only allow image files to be stored or if we are willing to allow arbitrary data. Also, another big question is whether to impose a maximum file size, say 100mb. 
Again, each Supernode would specify their minimum fee level that they charge per megabyte to store data in the storage layer (note that, while these Supernodes would receive the fee, ALL Supernodes would be equally responsible for storing the data based on the mechanics of how Kademlia works.) 
While the storage for Pastel Network NFT files is totally standardized in terms of the amount of RaptorQ symbol files in megabytes to generate for every megabyte of original data, and also in terms of how many times the same RaptorQ symbol file is stored inside Kademlia, it makes sense to let users of external Pastel storage select these parameters themselves. That way, users can decide if they want to spend more on storage to achieve a higher level of security/permanence or whether they are willing to take more of a risk of potential data loss if it means being able to save in PSL storage fees. All we need for this is to create a simple function for computing how the storage fees should be modified based on changes to these parameters; the example Python function below would work:
That is, the fees specified by the user as Maximum_PSL_Fee_for_Storage, and the fees specified by each Supernode per megabyte of storage, would both be for some “base level” of the two parameters  desired_rq_symbol_files_to_original_data_ratio and desired_redundancy_per_rq_symbol_file. If the user specified lower values for these two parameters (say, cutting each one in half), then the resulting PSL_Storage_Fee_Multiplier_Factor would also decrease. 
The total fee for storage would be found by first taking the Combined_Total_Storage_Fee_Per_Megabyte (computed by adding up the fees set by each of the 3 top-ranked Supernodes) as the “base fee” amount. Then, to compute what the user would actually pay, you would need to multiple by the PSL_Storage_Fee_Multiplier_Factor and the File_Size_in_Megabytes for the file selected by the user for storage. That is, as long as the following product: (Combined_Total_Storage_Fee_Per_Megabyte*PSL_Storage_Fee_Multiplier_Factor)*File_Size_in_Megabytes
is less than the value set by the user for Maximum_PSL_Fee_for_Storage, the External_Storage_Request will be processed by the 3 top-ranked Supernodes; if the product is not less than the Maximum_PSL_Fee_for_Storage set by the user, the request would remain pending until a set of 3 top-ranked Supernodes collectively accept a low enough fee so that it’s below the Maximum_PSL_Fee_for_Storage.
Then, once the user has paid this amount to the Supernodes from their External_Storage_Sending_Address (and also burned 20% of this amount), the Supernodes would each generate the required number of RaptorQ symbol files and would compare result to verify that the hashes of all of these RaptorQ symbol files matches between the 3 Supernodes, and that each Supernode could reconstruct the original file from these RaptorQ symbol files and get back the exact same SHA3-256 hash as the original file. 
Then these RaptorQ symbol files would be distributed across all the Supernodes using Kademlia in the usual manner. Also, the top-ranked Supernode would generate a JSON file containing the hashes of all of the RaptorQ symbol files and store that in Kademlia (actually, multiple different copies of this file would be generated and stored separately in Kademlia for additional redundancy). 
Finally, the 3 Supernodes responsible for the storage task would create an External_Storage_Registration_Ticket on the Pastel blockchain that summarizes the results of the storage operation. Essentially, this ticket would contain a single SHA3-256 hash, which would point to a JSON file in Kademlia containing the SHA3-256 hashes of all of the redundant JSON files stored in Kademlia, which themselves contain the SHA3-256 hashes of all of the RaptorQ symbol files generated by the storage operation. This ticket would be signed by each of the 3 Supernodes and by the user to activate it. 
The file could only be requested from the storage layer by the PastelID of the user who created the External_Storage_Request; the could do this by signing the request for the file with their PastelID, and Supernodes would only respond to valid signed requests. Otherwise, this external storage service could place excessively high bandwidth requirements on Supernodes and could be an attack vector for DDOS attempts.
To summarize, we would have two new data structures with the following fields:

1. External_Storage_Request (made via gRPC):
1.The PastelID of the user submitting the request.
2.The SHA3-256 hash of the file the user wants store in the storage layer.
3.The size of the file the user wants to store in megabytes. 
4.The Maximum_PSL_Fee_for_Storage.
5.The parameter desired_rq_symbol_files_to_original_data_ratio.
6.The parameter desired_redundancy_per_rq_symbol_file.
7.The resulting PSL_Storage_Fee_Multiplier_Factor.
8.The External_Storage_Sending_Address.
9.The Pastel signature of the user on the above data.
10.The Pastel signatures of the 3 top-ranked Supernodes at the time the ticket is submitted on the above data.
11.The SHA3-256 hash of all of the above data, which would serve as a unique identifier for the request, the Request_Identifier.
2.External_Storage_Registration_Ticket (made via a Pastel Blockchain Ticket):
1.The Request_Identifier of the corresponding External_Storage_Request.
2.The SHA3-256 hash of the file the user wants store in the storage layer.
3.The size of the file the user wants to store in megabytes. 
4.The Maximum_PSL_Fee_for_Storage.
5.The parameter desired_rq_symbol_files_to_original_data_ratio.
6.The parameter desired_redundancy_per_rq_symbol_file.
7.The resulting PSL_Storage_Fee_Multiplier_Factor.
8.The External_Storage_Sending_Address.
9.The effective total fee paid by the user in PSL (both paid to Supernodes and burned).
10.The SHA3-256 hash for the JSON file stored in Kademlia which itself contains the SHA3-256 hashes of the set of redundant copies of the JSON files in Kademlia (which contain the SHA3-256 hashes of all the RaptorQ symbol files).
11.The Pastel signatures of the Supernodes that generated the RaptorQ symbol files for the ticket on the above data.
12.The Pastel signature of the user on the above data.
13.The Pastel TXID of this transaction would then serve as the identifier for this ticket.
Sense Protocol - Previous
Sense Basics
Next - Cascade Protocol
Cascade Overview
Last modified 1yr ago
Export as PDF
Copy link
On this page
Background
Summary:
External_Sense_Request (made via gRPC):