Refactor core and extensions into separate namespaces

README.md

@@ -4,6 +4,9 @@ A minimal bitcoin library for MicroPython and Python3 with a focus on embedded s
 
 Should remain minimal to fit in a microcontroller. Also easy to audit.
 
+The core package stays focused on Bitcoin primitives and wallet formats. Optional
+features live under `embit.ext`, including Liquid support and SLIP-39.
+
 Examples can be found in [`examples/`](./examples) folder.
 
 Documentation: https://embit.rocks/
@@ -16,7 +19,9 @@ Support the project: `bc1qd4flfrxjctls9ya244u39hd67pcprhvka723gv`
 
 Requires a custom MicroPython build with extended [`hashlib`](https://github.com/diybitcoinhardware/f469-disco/tree/master/usermods/uhashlib) module and [`secp256k1`](https://github.com/diybitcoinhardware/secp256k1-embedded) bindings.
 
-To install copy the content of `embit` folder to the board. To save some space you can remove files `embit/util/ctypes_secp256k1.py` and `embit/util/pyhashlib.py` - they are used only in Python3.
+To install, copy the content of the `embit` folder to the board. For a core-only
+deployment you can omit `embit/ext`. To save some space you can also remove
+`embit/util/ctypes_secp256k1.py` - it is used only in Python3.
 
 ## Python 3
 
@@ -26,7 +31,7 @@ To install in development mode (editable) clone and run `pip3 install -e .` from
 
 PyPi installation includes prebuilt libraries for common platforms (win, macos, linux, raspi) - see [`src/embit/util/prebuilt/`](./src/embit/util/prebuilt/) folder. Library is built from [libsecp-zkp](https://github.com/ElementsProject/secp256k1-zkp) fork for Liquid support, but will work with pure [libsecp256k1](https://github.com/bitcoin-core/secp256k1) as well - just Liquid functionality doesn't work. If it fails to use the prebuilt or system library it will fallback to pure python implementation.
 
-If you want to build the lib yourself, see: [Building secp256k1 for `embit`](/secp256k1/README.md).
+If you want to build the lib yourself, see: [Building secp256k1 for `embit`](./secp256k1/README.md).
 
 
 ## Using non-English BIP39 wordlists
@@ -87,6 +92,13 @@ Run tests with desktop python:
 pytest
 ```
 
+Run only the core or extension suites:
+
+```sh
+pytest tests/core
+pytest tests/ext
+```
+
 Run tests with micropython:
 
 ```sh

docs/README.md

@@ -9,9 +9,9 @@ For cryptography it uses [libsecp256k1](https://github.com/bitcoin-core/secp256k
 - [BIP-39](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki), [BIP-32](https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki) key derivation (API docs: [bip39](./api/bip39.md), [bip32](./api/bip32.md))
 - parsing and signing [PSBT](https://github.com/bitcoin/bips/blob/master/bip-0174.mediawiki) transactions - both version 1 and 2 ([API docs](./api/psbt.md))
 - signing with custom SIGHASH flags
-- Descriptors and miniscript support ([API docs](./api/descriptor.md))
-- SLIP-39 Shamir Secret Sharing scheme (experimental, [API docs](./api/slip39.md))
-- Liquid network support (experimental, [API docs](./api/liquid/README.md))
+- Descriptors and miniscript support ([API docs](./api/descriptor/README.md))
+- SLIP-39 Shamir Secret Sharing scheme (`embit.ext.slip39`, experimental, [API docs](./api/ext/slip39.md))
+- Liquid network support (`embit.ext.liquid`, experimental, [API docs](./api/ext/liquid/README.md))
 - Taproot support (in progress, experimental)
 
 ## Installation
@@ -23,7 +23,10 @@ pip3 install embit
 
 **Micropython:** requires custom build with C bindings to hashlib and secp256k1. Docs TBD, see examples for now: [stm32](https://github.com/diybitcoinhardware/f469-disco), [RiscV](https://github.com/stepansnigirev/MaixPy), [esp32](https://github.com/stepansnigirev/esp32_embit)
 
-<!-- For more details check out [installation instructions](./install.md). -->
+The full Python package includes both core modules and optional extensions under
+`embit.ext`. Core-only MicroPython or vendored deployments can omit `embit/ext`.
+
+<!-- Installation details TBD. -->
 
 ## Basic usage
 

docs/api/README.md

@@ -45,7 +45,7 @@ Library is splitted into modules, list of modules sorted by topic:
 
 **Keys:**
 
-- [ec](./ec.md) - individual elliptic curve keys (`PrivateKey`, `PublicKey`) and signatures (`Signature`, `SchnorrSig`)
+- [ec](./ec/README.md) - individual elliptic curve keys (`PrivateKey`, `PublicKey`) and signatures (`Signature`, `SchnorrSig`)
 - [bip39](./bip39.md) - helper functions for mnemonics
 - [bip32](./bip32.md) - extended private and public keys (`HDKey`)
 
@@ -69,5 +69,5 @@ Library is splitted into modules, list of modules sorted by topic:
 - [hashes](./hashes.md) - common bitcoin hash functions like `tagged_hash`, `double_sha256` etc
 
 **Extensions / experimental:**
-- [liquid](./liquid/README.md) - support for confidential assets, liquid transactions, blinded descriptors and psets (elements version of psbt)
-- [slip39](./slip39.md) - shamir secret sharing scheme
+- [liquid](./ext/liquid/README.md) - `embit.ext.liquid` support for confidential assets, liquid transactions, blinded descriptors and psets (elements version of psbt)
+- [slip39](./ext/slip39.md) - `embit.ext.slip39` shamir secret sharing scheme

docs/api/_sidebar.md

@@ -16,6 +16,6 @@
 	- [bech32](/api/bech32.md)
 	- [compact](/api/compact.md)
 	- [hashes](/api/hashes.md)
-	- [liquid](/api/liquid/README.md)
-	- [slip39](/api/slip39.md)
-- [Recepies](/recepies/README.md)
+	- [liquid](/api/ext/liquid/README.md)
+	- [slip39](/api/ext/slip39.md)
+- [Recepies](/recepies/README.md)

docs/api/ec/_sidebar.md

@@ -20,6 +20,6 @@
 	- [bech32](/api/bech32.md)
 	- [compact](/api/compact.md)
 	- [hashes](/api/hashes.md)
-	- [liquid](/api/liquid/README.md)
-	- [slip39](/api/slip39.md)
-- [Recepies](/recepies/README.md)
+	- [liquid](/api/ext/liquid/README.md)
+	- [slip39](/api/ext/slip39.md)
+- [Recepies](/recepies/README.md)

docs/api/ec/public_key.md

@@ -125,7 +125,7 @@ Verifies Schnorr signature against the message.
 
 **Arguments**
 
-- `sig` - Instance of [`SchnorrSig`](./shcnorr_sig.md) to verify
+- `sig` - Instance of [`SchnorrSig`](./schnorr_sig.md) to verify
 - `msg` - `32`-byte hash of the message to verify against.
 
 **Returns**

docs/api/ext/liquid/README.md

@@ -0,0 +1,4 @@
+# Liquid network
+
+Liquid support is an optional extension. Import Liquid modules from
+`embit.ext.liquid`.

docs/api/ext/slip39.md

@@ -0,0 +1,3 @@
+# Shamir Secret Sharing scheme
+
+SLIP-39 support is an optional extension. Import it from `embit.ext.slip39`.

docs/api/networks.md

@@ -32,7 +32,7 @@ Here is a list of keys for `NETWORKS['main']`:
 }
 ```
 
-> If you are using Liquid you should also check out [`liquid.networks`](./liquid/networks.md) module.
+> If you are using Liquid you should also check out the `embit.ext.liquid.networks` module.
 
 If you want to add any bitcoin-compatible altcoin (doge, litecoin, whatever) - just add a new network dict to `embit.networks.NETWORKS`.
 
@@ -82,4 +82,4 @@ sc = script.p2wpkh(prv)
 print(sc.address(net))
 # >>> bcrt1qsvsxz8lsxg3rc86tk8am6g53l54n7s7eq98mfg
 
-```
+```

pyproject.toml

@@ -52,12 +52,13 @@ embit = ["util/prebuilt/*"]
 [tool.pytest.ini_options]
 addopts = "--cov=embit"
 testpaths = [
-    "tests/tests",
+    "tests/core",
+    "tests/ext",
 ]
 
 [tool.coverage.run]
 omit = [
-    "*/liquid/*",
+    "*/ext/liquid/*",
 ]
 
 [tool.mypy]
@@ -74,4 +75,3 @@ warn_return_any = true
 warn_unreachable = true
 warn_unused_configs = true
 no_implicit_reexport = false
-

src/embit/descriptor/miniscript.py

@@ -87,7 +87,7 @@ def wrapped(m_script):
     def read_arguments(cls, s, taproot=False):
         args = []
         if cls.NARGS is None:
-            if type(cls.ARGCLS) == tuple:
+            if isinstance(cls.ARGCLS, tuple):
                 firstcls, nextcls = cls.ARGCLS
             else:
                 firstcls, nextcls = cls.ARGCLS, cls.ARGCLS
@@ -901,7 +901,7 @@ def inner_compile(self):
 
     def __len__(self):
         return len(self.arg) + 1
-    
+
     def verify(self):
         super().verify()
         if self.arg.type != "V":

src/embit/ext/__init__.py

@@ -0,0 +1 @@
+

src/embit/liquid/addresses.py → src/embit/ext/liquid/addresses.py

@@ -1,4 +1,4 @@
-from .. import bech32, ec, script, base58
+from ... import bech32, ec, script, base58
 from . import blech32
 from .networks import NETWORKS
 

src/embit/liquid/blip32.py → src/embit/ext/liquid/blip32.py

@@ -1,6 +1,5 @@
 """BIP-32 for blinding keys. Non-standard yet!!!"""
-import sys
-from .. import bip32, ec
+from ... import bip32, ec
 from .networks import NETWORKS
 import hmac
 

src/embit/liquid/descriptor.py → src/embit/ext/liquid/descriptor.py

@@ -1,13 +1,13 @@
-from .. import ec
-from ..descriptor.descriptor import Descriptor
-from ..descriptor.base import DescriptorBase
-from ..descriptor.errors import DescriptorError
-from ..descriptor.arguments import Key
+from ... import ec
+from ...descriptor.descriptor import Descriptor
+from ...descriptor.base import DescriptorBase
+from ...descriptor.errors import DescriptorError
+from ...descriptor.arguments import Key
 from .networks import NETWORKS
 from .addresses import address
 from . import slip77
-from ..hashes import tagged_hash, sha256
-from ..misc import secp256k1
+from ...hashes import tagged_hash, sha256
+from ...misc import secp256k1
 
 
 class LDescriptor(Descriptor):

src/embit/liquid/finalizer.py → src/embit/ext/liquid/finalizer.py

@@ -1,6 +1,6 @@
-from .. import ec
-from ..script import Witness, Script
-from .transaction import TxOutWitness, Proof, LTransaction
+from ... import ec
+from ...script import Witness, Script
+from .transaction import LTransaction
 
 
 def parse_multisig(sc):

src/embit/liquid/networks.py → src/embit/ext/liquid/networks.py

@@ -1,4 +1,4 @@
-from .. import networks
+from ... import networks
 
 const = lambda x: x
 

src/embit/liquid/pset.py → src/embit/ext/liquid/pset.py

@@ -3,12 +3,10 @@
 if sys.implementation.name == "micropython":
     import secp256k1
 else:
-    from ..util import secp256k1
+    from ...util import secp256k1
 
-from .. import compact, hashes
-from ..psbt import *
-from collections import OrderedDict
-from io import BytesIO
+from ... import hashes
+from ...psbt import *
 from .transaction import (
     LTransaction,
     LTransactionOutput,
@@ -22,7 +20,8 @@
     unblind,
 )
 from . import slip77
-import hashlib, gc
+import hashlib
+import gc
 
 
 class LInputScope(InputScope):

src/embit/liquid/psetview.py → src/embit/ext/liquid/psetview.py

@@ -1,4 +1,4 @@
-from ..psbtview import *
+from ...psbtview import *
 from .pset import *
 import hashlib
 
@@ -232,7 +232,7 @@ def sighash_segwit(
         h.update(inp.sequence.to_bytes(4, "little"))
         if inp.has_issuance:
             inp.asset_issuance.hash_to(h)
-        if not (sh in [SIGHASH.NONE, SIGHASH.SINGLE]):
+        if sh not in [SIGHASH.NONE, SIGHASH.SINGLE]:
             h.update(hashlib.sha256(self.hash_outputs()).digest())
             if hash_rangeproofs:
                 h.update(hashlib.sha256(self.hash_rangeproofs()).digest())

src/embit/liquid/slip77.py → src/embit/ext/liquid/slip77.py

@@ -1,5 +1,5 @@
 import hmac
-from ..ec import PrivateKey
+from ...ec import PrivateKey
 
 DOMAIN = b"Symmetric key seed"
 LABEL = b"SLIP-0077"

src/embit/liquid/transaction.py → src/embit/ext/liquid/transaction.py

@@ -1,16 +1,14 @@
 import sys
-import io
-from .. import compact
-from ..script import Script, Witness
-from .. import hashes
-from ..transaction import *
-from ..base import EmbitBase
+from ... import compact
+from ...script import Script, Witness
+from ...transaction import *
+from ...base import EmbitBase
 import hashlib
 
 if sys.implementation.name == "micropython":
     import secp256k1
 else:
-    from ..util import secp256k1
+    from ...util import secp256k1
 
 
 class LSIGHASH(SIGHASH):
@@ -369,7 +367,7 @@ def sighash_segwit(
         h.update(inp.sequence.to_bytes(4, "little"))
         if inp.has_issuance:
             inp.asset_issuance.hash_to(h)
-        if not (sh in [SIGHASH.NONE, SIGHASH.SINGLE]):
+        if sh not in [SIGHASH.NONE, SIGHASH.SINGLE]:
             h.update(hashlib.sha256(self.hash_outputs()).digest())
             if hash_rangeproofs:
                 h.update(hashlib.sha256(self.hash_rangeproofs()).digest())

src/embit/slip39.py → src/embit/ext/slip39.py

@@ -1,7 +1,7 @@
 import hmac
 import hashlib
-from .bip39 import mnemonic_from_bytes, mnemonic_to_bytes
-from .misc import secure_randint
+from ..bip39 import mnemonic_from_bytes, mnemonic_to_bytes
+from ..misc import secure_randint
 from .wordlists.slip39 import SLIP39_WORDS
 
 

src/embit/ext/wordlists/__init__.py

@@ -0,0 +1 @@
+

src/embit/ext/wordlists/uslip39.py

@@ -0,0 +1,4 @@
+from ...wordlists.base import WordlistBase as _WordlistBase
+import uembit as _uembit
+
+SLIP39_WORDS = _WordlistBase(_uembit.ext.wordlists.slip39)

src/embit/misc.py

@@ -4,9 +4,9 @@
 # implementation-specific functions and libraries:
 if sys.implementation.name == "micropython":
     from micropython import const
-    import secp256k1
+    import secp256k1 as secp256k1
 else:
-    from .util import secp256k1
+    from .util import secp256k1 as secp256k1
 
     def const(x):
         return x

src/embit/transaction.py

@@ -282,7 +282,7 @@ def sighash_segwit(self, input_index, script_pubkey, value, sighash=SIGHASH.ALL)
         h.update(script_pubkey.serialize())
         h.update(int(value).to_bytes(8, "little"))
         h.update(inp.sequence.to_bytes(4, "little"))
-        if not (sh in [SIGHASH.NONE, SIGHASH.SINGLE]):
+        if sh not in [SIGHASH.NONE, SIGHASH.SINGLE]:
             h.update(hashlib.sha256(self.hash_outputs()).digest())
         elif sh == SIGHASH.SINGLE and input_index < len(self.vout):
             h.update(

src/embit/util/ctypes_secp256k1.py

@@ -1,4 +1,5 @@
-import ctypes, os
+import ctypes
+import os
 import ctypes.util
 import platform
 import threading
@@ -7,14 +8,12 @@
     cast,
     byref,
     c_char,
-    c_byte,
     c_int,
     c_uint,
     c_char_p,
     c_size_t,
     c_void_p,
     c_uint64,
-    create_string_buffer,
     CFUNCTYPE,
     POINTER,
 )
@@ -730,7 +729,7 @@ def _hashfn(out, x, y):
             y = ctypes.string_at(y, 32)
             try:
                 res = hashfn(x, y, data)
-            except Exception as e:
+            except Exception:
                 return 0
             out = cast(out, POINTER(c_char * 32))
             out.contents.value = res

src/embit/util/secp256k1.py

@@ -1,6 +1,5 @@
 try:
     # if it's micropython
-    from micropython import const
     from secp256k1 import *
 except:
     # we are in python