How to remove file types from a folder of 10000 json files and another folder of 10000 png files

As part of a 10k NFT project, how can I upload a folder of 10000 files to IPFS that does not contain the file type.

What I mean is that I want the tokenURI to be able to point to ipfs//HASH/123 instead of ipfs//HASH/123.json

I see projects like Bored APE Yacht Club and Sneaky Vampires accomplish this, yet when I have tried (using Pinata to upload and pin) then it does not properly resolve and display the JSON file in the browser.

Here is an example of Sneaky Vampire, and you can see their json file does not have .json at the end of it: https://gateway.ipfs.io/ipfs/bafybeic26wp7ck2bsjhjm5pcdigxqebnthqrmugsygxj5fov2r2qwhxyqu/12

When I try the json files do not resolve and display in the browser, instead the browser just downloads a file: https://gateway.ipfs.io/ipfs/QmYvzDwfTagHZfLazRrxosdc4PVH2bBBFBuzyWEzzdhmkR/12

It works, your browser is too smart, use dumb tools instead

I can download the file perfectly well.
You are on a wrong track.
Yes there is an issue however it’s with mime type deduction. Simply whatever the gateway use belive it’s an DBF file (database used by some spreadsheet software) instead of a json one.
This is harmless and your json is fine.

You are using a much more intelligent piece of software (a browser) to view the file that need to first figure out what the file is before displaying it to you.
Using a CLI tool like ipfs cat or curling the gateway would have been a better idea as you would have seen the true nature of the file.

CRLF → LF

Peeking at your json I see that you are using CR LF (windows) encoding, instead of the recomended LF (*nix), to me that only obvious thing that a mime type detector would screw up (I might be wrong but your json is really fine).

You can fix that by running a linux VM (or WSL2, it’s probably fine but I like VMs more (no actually I like running linux on baremetal as it’s faster and I hate the buggy bloated shit that windows is but whatever)) and working in that unix friendly environment.
I don’t know what tool you are using, you can probably configure it to output LF lines, just google <Insert the software you use> output LF files.
Or lastly you can use a converter tool by googling “crlf to lf” and using whatever CLI tool stackoverflow propose you after 3 minutes of search.

On an other subject

Your metadata file is broken

Also I see your json is :


{
	"name": " #12",
	"description": "test description",
	"image": "ipfs/QmXtr1NjrvafSM2NkzBqEwh8E6uCFFDVGv46uCrf1FXMG1/image12.png",
	"attributes":[
		{
			"trait_type": "Background",
			"value": "Green"
		},
		{
			"trait_type": "Shape",
			"value": "circle"
		},
		{
			"trait_type": "Emotion",
			"value": "sad"
		}
	]
}

The image field is wrong, it must be ipfs://... so for yours ipfs://QmXtr1NjrvafSM2NkzBqEwh8E6uCFFDVGv46uCrf1FXMG1/image12.png.

Consider minifying it

Also as a one last thing.
This is meant to be red by machines, not humans.
So you should minifiy it (remove useless indentation and new lines) like this:

{"name":" #12","description":"test description","image":"ipfs://QmXtr1NjrvafSM2NkzBqEwh8E6uCFFDVGv46uCrf1FXMG1/image12.png","attributes":[{"trait_type":"Background","value":"Green"},{"trait_type":"Shape","value":"circle"},{"trait_type":"Emotion","value":"sad"}]}

This also increase chances of success because in case you ever encounter a decoder that only supports one type of new lines, well there is no newlines, everything is compacted together so no issue can occur. :slight_smile:

2 Likes

Thank you so much for the detailed response!
I generated the files on my windows laptop using a python script in vscode and a powershell terminal. I had no idea about the difference between CRLF and LF, so appreciate that. Maybe one day I will actually learn Linux, though thanks for giving me the tips on what to search for in the meantime and for the catch in the image field!

By the way, I found out that changing CRLF to LF in VSCode just involves clicking a button in the lower right corner of window
image

I belive doing this change your python script itself.
But not the files produced by python script.

Yeah, it’s really that windows use two characters for a new line, CR (Cariage Return) and LF (Line Feed), while almost everything else only use LF.

A while ago, at a time where terminals where the main human machine interface CR vs LF might have been a good idea because you could only LF so you would imagine your cursor going down a line but not snapping back to the left of your screen too (like how the arrow keys works right now) (or just CR and snapping left without going down).
But I havn’t found lots of software that still supports this (I only know of the windows CLI) and CRLF is 2 bytes instead of 1.

1 Like

I belive you need to add newline="\n" to your open call.

Like this : How to convert CRLF to LF on a Windows machine in Python - Stack Overflow

1 Like

IT WORKS!!!

https://gateway.ipfs.io/ipfs/QmdrBa1X56v4beBBHYrG1m2X2rHK3UnorSLSEiWHJeN4rj/12

I feel so much better now and actually understand it now, thanks!