Adding string or json object larger than 4096 characters doesn't work anymore

Help
1

When i add a string or a json object that is larger than 4096 characters and then try to retrieve it, the retrieved string is cut off at 4096 characters.

Has anything changed regarding the max size of a string or json object that can be stored on ipfs?
This used to work for me in the past, but not anymore. I did recently upgrade to the new ipfshttpclient for python 3.

2

(not familiar with ipfshttpclient for python)

How are you adding it? How are you retrieving it?

You should be able to add arbitrarily large objects split into DAG structures, but I thought there were limits on the size of each individual DAG block. I think that DAG nodes were limited to around 2MB each, but that may not be the latest. Ideally, it would make sense to use APIs that automatically take care of chunking and creating appropriate DAGs.

5

Here is some code to reproduce the issue:

import ipfshttpclient

api = ipfshttpclient.connect(‘/ip4/127.0.0.1/tcp/5001/http’)

length = 4097 # up to 4096 should work

original_string = ‘’.join([‘a’ for _ in range(length)])
multihash = api.add_str(original_string)

retrieved_string = api.cat(cid=multihash)

retrieved_string = retrieved_string.decode() # In python3, the retrieved string is a bytes object, so need to decode it back into a string

print(original_string)
print(retrieved_string)
assert original_string == retrieved_string

6

Does this look related to this open issue with for ipfshttpclient? It seems similar to me.

It looks like the py-ipfs-http-client only supports go-ipfs versions up to 0.4.19 (with newer versions having “compatiblity problems” [sic]). So, if you’re using the latest version (latest is 0.4.22) I wonder if this could be one of the compatibility problems.

7

Hm. There was an issue < 0.4.19 where files could be truncated on upload due to a bug in the go http library. However, I’m not aware of any issues in go-ipfs >= 0.4.19 that could have caused this.

There was an issue adding multiple files but that was fixed in 0.4.21 and shouldn’t apply to single files.

1 Like
8

@leerspace Yes, it does seem similar to that open issue

i’m using :
Python 3.7
go-ipfs 0.4.22
ipfshttpclient 0.4.12

I tried downgrading the go-ipfs to several earlier versions, versions 19, 21 and 22 all have the same behavior, 18 and 20 threw compatibility errors

9

seems there is another open issue regarding this problem, it mentions setting the chunk-size while connecting to the ipfs node.

I have tried setting chunk size to a larger number and this does seem to work.
However I’m not sure if this could have any side effects if i set it to a very large number.

client = ipfshttpclient.connect('/ip4/127.0.0.1/tcp/5001/http"', chunk_size=10000)

https://github.com/ipfs/py-ipfs-http-client/issues/190

10

found the problem and created a pull request:

1 Like