[Proposal] Lightweight Python Runtime SDK


#1

Problem Statement
TL;DR: Writing scripts to interact with zend has a high barrier of entry which would benefit from being lowered.

During routine management of multiple secure nodes for multiple users, including charging VPS costs + management fees and generate a P&L sheet for my customers, I wanted to automate away a lot of the transactions and documentation work. I ended up writing a large bash script due to the python-based tools in the codebase being clunky and inaccessible. If there had been an easy-to-use, well documented python SDK, I could have written/tested the automation scripts an order of magnitude faster and would feel much better about needing to automate more tasks in the future.

Solution
Create a lightweight python sdk for management and scripting. Can be integrated directly into the zen repo or in its own. Needs to be stable, 1-to-1 to zen-cli, well tested, but most of all incredibly well documented (so it is easy to use, the whole point of this proposal is to lower barrier of entry). This is a solution of 2 parts to lower the technical barrier of entry to using the zencash system:

  1. Easy to use python SDK (code)
  2. Well written documentation (understanding)*

*This documentation will be written in a portable way to copy over to wikis for both the zen-cli and python SDKs (since they will be 1-to-1)

About Me
I am a software developer based out of San Jose, CA currently working as a cloud network security engineer for a small startup. Due to the company being small I also handle all of our technical documentation, QA, Product Management and Customer Support. I can provide samples of all of the above on request. Past experience:

  • BS Computer Science from University of California: Irvine. Specializations in Low-Level Systems and Machine Learning
  • Undergraduate Researcher doing GPU acceleration refactoring on 3 separate projects: SANA (simulated annealing for network alignment: network alignment using simulated annealing to find similarities between protein-based lifeforms on a molecular level), ISSM (Ice Sheet System Model: ice sheet system modeling to simulate global warming) and SpaRc FiRe (Spiral Arc Finder and Reporter for finding spiral galaxies in the backgrounds of old hubble images).
  • Product Management at Violin Memory - Created tools to expedite communication channels between the engineering and business sides of the company: Machine Learning tool to generate quotes for potential customers based on past data, their size, networking and storage needs. Ticketing tool to distribute company sports game tickets leftover from sales. Benchmark visibility tool to generate benchmark graphs on demand given certain parameter sets.
  • Developer at Ring - Employee 5, created website, backend, and order fulfillment system. Now a 1000+ employee company looking to IPO next year
  • Marketing at Vidtel - Handled multiple marketing campaigns. Also refactored the website from scratch when needed.
    …etc

LinkedIn: https://www.linkedin.com/in/colin-mcgrath-41974253/
Can provide resume and more info on request.

Outside of work I enjoy building computers (mining rigs included!), surfing, longboarding, golfing, going to concerts, reading, and most of all learning about new things!

Cost

See proposals for full breakdown. Nothing but testate coins due on acceptance. Most of cost is withheld until full project delivery.

  1. Version 1.0: 410 ZEN
  2. Version 1.1: 395 ZEN (after changes recommended by @kendrick: use Travis instead of Jenkins [free VPS])
  3. Version 2.0: 330 ZEN (after changes recommended by @Persona: no CI/QA, fork off python-bitcoinrpc)

Link to Proposal Versions


#2

Hey @cpmcgrat as a python developer myself, I am excited to see this proposal.

Anytime there are high cost proposals (410 Zen is around 17 thousand USD) I like to dig in and make sure our community is getting the best value for the price. Since zencash is ultimately a fork of zcash and bitcoin, I did a bit of research to see what tools already existed in those communities that could be ported over. This way we minimize the amount of double work and fully utilize what has already been created.

It turns out the python-bitcoinrpc package integrates out of the box with zencash. You just have to make sure you have rpcuser, rpcpassword, rpcport, rpcallowip, and server configured correctly in the zen.conf file located in .zen folder on the node.

From the handful of commands I tested, it appears to work on a 1-to-1 with the zen-cli. So even though it was originally developed for bitcoin, the ZCash/ZenCash API commands work too.

pip install python-bitcoinrpc

In python

from bitcoinrpc.authproxy import AuthServiceProxy, JSONRPCException

rpc_connection = AuthServiceProxy("http://%s:%[email protected]:8231"%(rpc_user, rpc_password))
request1 = rpc_connection.z_listaddresses()
print(rpc_connection.getblock(request1))

Command parameters can be passed inside the parentheses of the request. The above would need to be executed on the same server as the node because I believe exposing the RPC calls anywhere other than localhost could be a security vulnerability, especially for secure nodes.

Now I’m not sure if you planned an implementation with the same solution but given that the tools already exist, part 1 of your proposal could essentially be a fork of the python-bitcoinrpc repo.

In light of this, my input as a community member on this proposal would be to revise the deliverables:

  1. Consider forking the python-bitcoinrpc and using it as the code base.
  2. Test each zen-cli command to:
    a) ensure each command works
    b) document how each command is used
  3. Register to pip and provide open source on github with shared access with zenofficial (so the zen dev team has the ability make future developments and pip deployments)
  4. Include package documentation in the readme or wiki on the github.

When looking through the other itemized costs, I don’t think this proposal needs continuous integration and quality assurance because the zen-cli is very rarely changed. I’m also unsure what the process manager is or what its role is in the proposal.

Thanks for reading

TL;DR I agree that having a well documented, easily accessible through pip, and zencash focused python package for the node RPC is important. We should make sure to utilize existing developments from the bitcoin/zcash communities. We should also exclude overkill features such as continuous integration. Lastly, I’m unsure what Process Manager and Server Runtime are or if they are needed.


#3

Hi @Persona, thank you for the detailed reply!

I do agree that the existing python-bitcoinrpc package you found makes a couple of the steps in my proposal redundant. I will go ahead and create a new version of my proposal to factor in using a fork of python-bitcoinrpc instead of writing much of the same code from scratch w/o a CI/QA system. I’ll leave the old proposal around with a [DEPRECIATED] tag for context.

I agree the CI/QA is heavy handed for the use case. However, my work philosophy is write tests for EVERYTHING so if someone else wants to pick up development, I don’t need to be around to explain the project to them. Maybe the Jenkins CI/QA system could be replaced by some inline unit tests and build/push scripts to impart the same contextual awareness?

As for your questions:

  • Server Runtime Cost - Cost of running VPS for Jenkins server (most likely no longer needed due to heavy-handed nature of approach)
  • Process Manager - Python module that nannies the zend process while the user is running their python scripts. This is so the end user can start/stop the zend process from python natively inside their script. It will be optional to use the process manager if you already have zend running in the background (the SDK will have a similar localhost API proxy method to bitcoinrpc), but I figured it would be nice to have a python-native way to manage the zend process.

#4

I totally agree with a CI/QA system. But why a Jenkins server, what benefits does the Jenkins server provide as opposed to the free alternative - Travis?


#5

Hi @kendrick, the Jenkins software stack itself is open-source and free. However, the cost of the VPS to run Jenkins + the IT management overhead is not.


#6

Hi @Persona,

I went ahead and made a refactored proposal following your comments. Total cost was reduced from 410 ZEN -> 330 ZEN with removal of CI/QA system and base from python-bitcoinrpc. Most of the reduction comes from the removal of the CI/QA system. I reduced the SDK creation cost by 10 ZEN since the majority of the work of supporting each zen-cli call as a class method instead of batched command string remains.

Please let me know your thoughts. I think once there is some engagement from either Rob or Rolf it may be good to put out a more detailed requirements doc for what the user interaction should look like.

Thank you!


#7

@cpmcgrat, I know that Jenkins is FOSS, I asking why not just use Travis which is free as long as your project is open sourced, as opposed needing to host a VPS and the additional IT management overhead?


#8

@kendrick, ah got it. I didn’t know they handled the VPS for you as well. That’s definitely a better option! Let me make a version of the proposal with that change in mind.


#9

@kendrick went ahead and made a new version of the proposal with your comments. Was unsure how you felt wrt the fork off python-bitcoinrpc so for now I didn’t include it. Please let me know your thoughts and I’d be happy to refactor it again. Current cost with your proposed changes is 395 ZEN (-15 ZEN for free managed VPS over 3 months)


#10

Thanks for both the proposal and the community contributions to the proposal. Let me take a look and discuss further with team.


#11

@Blockops Thank you! Let me know what you guys come up with.


#12

@cpmcgrat The developers on the team like the idea of Python SDK proposal you made. I’ve talked about it internally with a few different people and the consensus is to go ahead with your proposal.

My understanding is 330 ZEN, paid to the delivery milestones outlined in your proposal. As a point of clarification, I’d like to verify that the development work will be done in a repository on the ZenCashOfficial github, with regular commits as progress is made.

If that all sounds good, your proposal is accepted, and you can start on this as soon as you are able to.


#13

@Blockops sure thing, I can definitely develop on the ZenCashOfficial repository. I’ll open a new branch and make PRs for the milestones (I’ll update the proposal list with the accepted version with this change).

To confirm, you are accepting the version of the proposal amended by @Persona with no Jenkins-based CI/QA system (replaced by unit tests) and a fork off the python-bitcoinrpc system, correct?


#14

Yes, that is correct. Thank you!