Awesome Endeavour: DHT Part II by daviddias · Pull Request #856 · ipfs/js-ipfs

daviddias · 2017-05-19T14:54:58Z

This PR is the second part of the DHT integration in js-ipfs

Remaining tasks

Main work tracker: libp2p/js-libp2p-kad-dht#1

Needs the following PRs:

Blocked PRs:

daviddias · 2017-07-20T05:45:19Z

Peer and Content Routing (DHT) js-ipfs-bitswap#142

daviddias · 2017-08-25T15:25:03Z

It might actually be good to enable the DHT if it proves to not break things and then figure out the interop of the DHT with go-ipfs as we go.

daviddias · 2017-08-26T14:31:57Z

Updated deps including all the latest changes in Bitswap. DHT tests in JS land are fine. Still no love between JS and Go 😢

daviddias · 2017-08-26T14:32:07Z

CI is still not happy as well


  1) interface-ipfs-core tests .dht callback API .findpeer finds other peers:
     Uncaught AssertionError: expected [Error] to not exist
      at nodeA.dht.findpeer (node_modules/interface-ipfs-core/src/dht.js:90:32)
      at self._libp2pNode.peerRouting.findPeer (src/core/components/dht.js:81:18)
      at node_modules/async/internal/once.js:12:16
      at next (node_modules/async/waterfall.js:21:29)
      at node_modules/async/internal/onlyOnce.js:12:16
      at waterfall (node_modules/libp2p-kad-dht/src/index.js:449:20)
      at nextTask (node_modules/async/waterfall.js:16:14)
      at next (node_modules/async/waterfall.js:23:9)
      at node_modules/async/internal/onlyOnce.js:12:16
      at setImmediate (node_modules/multihashing-async/src/utils.js:8:7)
      at Immediate.<anonymous> (node_modules/async/internal/setImmediate.js:27:16)

  2) interface-ipfs-core tests .dht callback API .query returns the other node in the query:
     Uncaught AssertionError: expected [] to include 'QmS72d5n39h2zWXHQFFQs2Xub7GADbYimayvJYkocJv98D'
      at nodeA.dht.query (node_modules/interface-ipfs-core/src/dht.js:139:47)
      at self._libp2pNode._dht.getClosestPeers (src/core/components/dht.js:157:9)
      at q.run (node_modules/libp2p-kad-dht/src/index.js:172:18)
      at Query.run (node_modules/libp2p-kad-dht/src/query.js:47:14)
      at utils.convertBuffer (node_modules/libp2p-kad-dht/src/index.js:166:9)
      at setImmediate (node_modules/multihashing-async/src/utils.js:8:7)
      at Immediate.<anonymous> (node_modules/async/internal/setImmediate.js:27:16)

hoffmabc · 2018-02-16T15:28:19Z

Is this PR still up to date?

daviddias · 2018-02-16T17:11:03Z

@hoffmabc it is as with regards to status, DHT is still a WIP. Lot's of work happening on js-libp2p land itself -- https://github.com/libp2p/js-libp2p --. Currently focused on libp2p/js-libp2p#159

alanshaw · 2018-04-11T11:39:31Z

Is there any prior docs I can read about the technical challenges involved in go <-> js DHT interop?

daviddias · 2018-04-30T19:18:41Z

Let's unblock this endeavour with the delegated routing as soon as 0.29.0 gets released.

rmisio · 2019-01-03T14:19:19Z

@BazaarDog Those seem to be links to different tests. Does that mean that this PR here should already have the go interop baked in and just the tests are seperate?

alanshaw · 2019-01-03T16:46:30Z

@rmisio this PR should be interoperable with go-ipfs

rmisio · 2019-01-03T22:54:19Z

I can't seem to get the dht working right between two ipfs-js nodes:

// package.json
"ipfs": "github:ipfs/js-ipfs#feat/dht-part-ii",

// node config
{
      EXPERIMENTAL: {
        pubsub: true,
        dht: true,
      },
      relay: {
        "enabled": true,
      },
      repo: `./ipfs/${peerId}`,
      init: { privateKey },
    }

// receiving node puts item in the dht keyed by their peerId
        node.on('ready', () => {
          node.libp2p.start(() => {
            const val = `${peerId} - slick willy willy`;

            node._libp2pNode.dht.put(
              identity.peerId,
              Buffer.from(val),
              err => {
                if (err) {
                  throw err;
                }

                console.log(`"${val}" has been stored in the dht. Kudos.`);
              }
            );

I could verify that the node that puts the item in the dht is able to retrieve it.

But, a different node cannot:

    this.node._libp2pNode.dht.get(
      createFromB58String(receiver).id,
      {},
      (err, content) => {
        if (err) {
          throw err;
        }

        console.log(`for ${receiver} the dht has "${content.toString()}"`);
      }
    );

The main error I seem to get is Callback function "anonymous" timed out.:

I can confirm via node._libp2pNode.stats.peers() I have plenty of peers at the time of the dht.get call... often times over 90.

FWIW, the web nodes I'm using are each connected to a ipfs-go node acting as a relay. Not sure if that has anything to do with anything here :)

joeykrug · 2019-01-12T20:45:14Z

What's the status of getting this merged + functional? Looking to use this for faster sync by sharing db's over ipfs for AugurProject/augur#771

vasco-santos · 2019-01-12T21:44:18Z

Hey all!

This PR is interoperable with the go-ipfs DHT implementation. I intend to continue the work on this PR after the 21st January, in order to get this finished and merged.

daviddias · 2019-01-28T18:44:43Z

README.md

-**Please read this:** DHT (automatic content discovery) and Circuit Relay (pierce through NATs and dial between any node in the network) are two fundamental pieces that are not finalized yet. There are multiple applications that can be built without these two services but nevertheless they are fundamental to get that magic IPFS experience. If you want to track progress or contribute, please follow:
-
- DHT: https://github.com/ipfs/js-ipfs/pull/856
- ✅ Relay: https://github.com/ipfs/js-ipfs/pull/1063


daviddias · 2019-01-28T18:45:14Z

README.md

 - `pubsub` (boolean): Enable libp2p pub-sub. (Default: `false`)
 - `ipnsPubsub` (boolean): Enable pub-sub on IPNS. (Default: `false`)
 - `sharding` (boolean): Enable directory sharding. Directories that have many child objects will be represented by multiple DAG nodes instead of just one. It can improve lookup performance when a directory has several thousand files or more. (Default: `false`)
- `dht` (boolean): Enable KadDHT. **This is currently not interoperable with `go-ipfs`.**


Will the DHT start automatically in the browser too?

In the current state of this PR it is starting automatically in both browser and node. Do you think we should start by having only node.js nodes first?

I would recommend to start with node only and look at adding it to the browser as a default later, if ever. Until we get more performance benchmarking for the browser I'm hesitant to just turn it on there for everyone.

yeah, that's a good point!

OK, so, I'm half inclined to just enable it in the browser by default, expose ourselves to the issues and get them fixed asap. That said, since this is such a big feature perhaps not having it enabled by default in the browser initially will give us some scope to test it out first and fix the bigger issues, so that we don't expose all our users to the bugs and make everyone mad.

We do need a way to enable/disable it though - if the dht config option is being removed from EXPERIMENTAL, is it being added to the top level?

daviddias · 2019-01-28T18:46:09Z

src/cli/commands/dht.js

+
+  handler (argv) {
+  }
+}


Is the DHT full implemented on http api and CLI as well? Mind updating https://github.com/ipfs/ipfs/blob/master/IMPLEMENTATION_STATUS.md#dht ?

done in ipfs/ipfs#388

daviddias · 2019-01-28T18:47:03Z

src/core/components/dht.js

+        } catch (err) {
+          log.error(err)
+
+          return setImmediate(() => callback(errcode(err, 'ERR_INVALID_CID')))


Wasn't there a push to change setImmediate for nextTick?

daviddias · 2019-01-28T18:47:11Z

src/core/components/dht.js

-      if (typeof opts === 'function') {
-        callback = opts
-        opts = {}
+          return setImmediate(() => callback(errcode(err, 'ERR_INVALID_CID')))


daviddias · 2019-01-28T18:47:55Z

src/core/components/libp2p.js

      },
      EXPERIMENTAL: {
-        dht: get(options, 'EXPERIMENTAL.dht', false),
+        dht: !(get(options, 'local', false)),


What does this mean?

We have an option (local currently) that aims to:

Run offline. Do not connect to the rest of the network but provide local API.

Regarding this, go-ipfs changed recently this option name to offline. So, I created a PR now for renaming this option to offline #1850.

Once it is merged, I will rebase this PR and, if we go with DHT enabled by default in libp2p config, I will change this logic to not add the dht to the libp2p modules, if we are running offline.

daviddias · 2019-01-28T18:48:15Z

src/core/runtime/libp2p-browser.js

+        },
        EXPERIMENTAL: {
-          dht: false,
+          dht: true,


No need anymore, right?

In the libp2p side, it is still marked as experimental. I am totally up to make it enabled by default and running if libp2p receives the DHT in the modules object.

Should we release a new version of libp2p removing the experimental flag now? If you all agree, I will make the PR later today.

cc @jacobheun

Should we release a new version of libp2p removing the experimental flag now? If you all agree, I will make the PR later today.

I agree 👍 💯

I also agree, set it free!

daviddias · 2019-01-28T18:48:22Z

src/core/runtime/libp2p-nodejs.js

        },
        EXPERIMENTAL: {
-          dht: false,
+          dht: true,


No need anymore, right?

daviddias · 2019-01-28T18:49:00Z

src/http/api/resources/dht.js

+        }).code(500)
+      }
+
+      reply({})


Why send back an empty objective?

alanshaw

Additionally the CLI command handlers may soon need to be updated to use getIpfs instead of ipfs. See #1860 for details.

alanshaw · 2019-02-05T16:23:17Z

README.md

 - `pubsub` (boolean): Enable libp2p pub-sub. (Default: `false`)
 - `ipnsPubsub` (boolean): Enable pub-sub on IPNS. (Default: `false`)
 - `sharding` (boolean): Enable directory sharding. Directories that have many child objects will be represented by multiple DAG nodes instead of just one. It can improve lookup performance when a directory has several thousand files or more. (Default: `false`)
- `dht` (boolean): Enable KadDHT. **This is currently not interoperable with `go-ipfs`.**


OK, so, I'm half inclined to just enable it in the browser by default, expose ourselves to the issues and get them fixed asap. That said, since this is such a big feature perhaps not having it enabled by default in the browser initially will give us some scope to test it out first and fix the bigger issues, so that we don't expose all our users to the bugs and make everyone mad.

We do need a way to enable/disable it though - if the dht config option is being removed from EXPERIMENTAL, is it being added to the top level?

alanshaw · 2019-02-05T16:32:13Z

src/cli/commands/dht/find-peer.js

+      const peers = await ipfs.dht.findPeer(peerID)
+      const addresses = peers.multiaddrs.toArray().map((ma) => ma.toString())
+
+      print(addresses)


print is just console.log which will give you output like:

[ '/ip4/127.0.0.1/tcp/4001', '/ip4/169.254.83.88/tcp/4001', '/ip4/192.168.1.95/tcp/4001' ]

In comparison go-ipfs outputs:

$ ipfs dht findpeer QmZMxNdpMkewiVZLMRxaNxUeZpDUb34pWjZ1kZvsd16Zic /ip6/2001:19f0:5:66d1::/tcp/4001 /ip4/149.28.228.162/tcp/33154 /ip4/45.77.150.82/tcp/4001 /ip6/::1/tcp/4001 /ip6/2001:19f0:5:5d40:5400:1ff:fec1:ce94/tcp/4001 /ip4/127.0.0.1/tcp/4001 /ip4/149.28.228.162/tcp/4001 /ip4/127.0.0.1/tcp/8081/ws

alanshaw · 2019-02-05T16:33:50Z

src/cli/commands/dht/provide.js

+  },
+
+  handler ({ ipfs, key, resolve }) {
+    // TODO add recursive option


Please pass this option to ipfs.dht.provide so it can throw and inform the user the option is not implemented yet.

alanshaw · 2019-02-05T16:39:42Z

src/core/components/libp2p.js

      },
      dht: {
+        kBucketSize: get(options, 'dht.kBucketSize', 20),
+        enabled: get(options, 'dht.enabled', true) && !(get(options, 'local', false)),


"local" is now "offline"?

alanshaw · 2019-02-05T16:40:47Z

src/core/components/pin-set.js

-const some = require('async/some')
-const eachOf = require('async/eachOf')
+const someSeries = require('async/someSeries')
+const eachOfSeries = require('async/eachOfSeries')


What does this change do?

Using some is making a lot of parallel dht find and connect requests.

So, everything will get queried even if we find the target node, as the find takes longer than initiating the requests.

In the long run, we should refactor the pin-set, in order to try to parallelize some operations, but this is the safest option currently.

alanshaw · 2019-02-05T16:41:47Z

src/http/api/resources/dht.js

+      arg: Joi.string().required()
+    }).unknown()
+  },
+  handler: (request, reply) => {


These handlers now need to change for Hapi 18 - please shout if you have any questions!

daviddias added awesome endeavour status/in-progress In progress labels May 19, 2017

daviddias mentioned this pull request Jun 13, 2017

Site refactor libp2p/website#52

Merged

33 tasks

daviddias added status/ready Ready to be worked and removed status/in-progress In progress labels Jul 3, 2017

This was referenced Jul 5, 2017

Uploaded file via Spawned Node not accessible via public gateway #886

Closed

Implement IPNS #209

Closed

daviddias mentioned this pull request Aug 25, 2017

The file added by node.js can't be get/cat on other nodes #957

Closed

daviddias force-pushed the feat/dht-part-ii branch from 6a29c16 to df424e5 Compare August 26, 2017 14:03

ReisenB mentioned this pull request Sep 15, 2017

State of IPFS DistributedMemetics/DM#1

Open

daviddias added P0 Critical: Tackled by core team ASAP status/ready Ready to be worked and removed status/ready Ready to be worked labels Oct 17, 2017

daviddias mentioned this pull request Nov 11, 2017

Talking to js-ipfs richardschneider/net-ipfs-http-client#19

Open

RangerMauve mentioned this pull request Jan 9, 2018

pub/sub - publish / subscribe ipfs/notes#64

Open

3 tasks

dryajov mentioned this pull request Mar 26, 2018

OKR 2018 Q2 #1281

Merged

neuronsupport mentioned this pull request Apr 15, 2018

WebTorrent DHT integration webtorrent/webtorrent#1152

Open

daviddias mentioned this pull request Apr 23, 2018

feat: ping #1299

Merged

2 tasks

fazo96 mentioned this pull request May 2, 2018

peers don't connect automatically through circuit, but they do through websocket-star rendezvous server #1309

Closed

vasco-santos mentioned this pull request May 8, 2018

Merge branch 'master' of linonetwo/ipfs-browser-gateway ipfs-shipyard/service-worker-gateway-2019-poc#1

Closed

daviddias self-assigned this May 30, 2018

daviddias force-pushed the feat/dht-part-ii branch from df424e5 to 5347ffd Compare May 30, 2018 09:30

joeykrug mentioned this pull request Jan 13, 2019

Spike: Research whether to use webtorrent or IPFS AugurProject/augur#771

Closed

alanshaw mentioned this pull request Jan 17, 2019

⚡️ v0.35.0 RELEASE 🚀 #1826

Closed

50 tasks

This was referenced Jan 21, 2019

fix: race condition causing Database is not open error #1834

Merged

fix: ipv6 naming with multiaddr-to-uri package libp2p/js-libp2p-websockets#81

Merged

alanshaw mentioned this pull request Jan 24, 2019

[WIP] test: add ipns dht tests ipfs/interop#47

Open

3 tasks

daviddias commented Jan 28, 2019

View reviewed changes

This was referenced Jan 29, 2019

chore: update dht implementation status ipfs/ipfs#388

Merged

chore: enable dht by default libp2p/js-libp2p#313

Merged

alanshaw suggested changes Feb 5, 2019

View reviewed changes

alanshaw mentioned this pull request Feb 6, 2019

dht functionality #1857

Closed

Alan Shaw and others added 9 commits February 6, 2019 15:41

chore: increase silent timeout even more

53a3f32

feat: dht ready

28f5641

test: enable dht core test

8ecb7b5

feat: add dht cli

f9cd739

chore: enable dht by default

eadd7d5

fix: code review

4453b55

fix: code review

2a76f98

fix: ci tests

045e305

fix: code review

2956637

alanshaw approved these changes Feb 6, 2019

View reviewed changes

Alan Shaw and others added 2 commits February 7, 2019 11:49

Merge branch 'master' into feat/dht-part-ii

2651787

fix: make ci happy again

02d1704

vasco-santos mentioned this pull request Feb 8, 2019

feat: limit connections number #1872

Merged

4 tasks

Conversation

daviddias commented May 19, 2017 • edited by vasco-santos Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Remaining tasks

Needs the following PRs:

Blocked PRs:

Uh oh!

daviddias commented Jul 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daviddias commented Aug 25, 2017

Uh oh!

daviddias commented Aug 26, 2017

Uh oh!

daviddias commented Aug 26, 2017

Uh oh!

hoffmabc commented Feb 16, 2018

Uh oh!

daviddias commented Feb 16, 2018

Uh oh!

alanshaw commented Apr 11, 2018

Uh oh!

daviddias commented Apr 30, 2018

Uh oh!

rmisio commented Jan 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alanshaw commented Jan 3, 2019

Uh oh!

rmisio commented Jan 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joeykrug commented Jan 12, 2019

Uh oh!

vasco-santos commented Jan 12, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alanshaw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

daviddias commented May 19, 2017 •

edited by vasco-santos

Loading

daviddias commented Jul 20, 2017 •

edited

Loading

rmisio commented Jan 3, 2019 •

edited

Loading

rmisio commented Jan 3, 2019 •

edited

Loading