Python : 100 days from Nob to Master

Real_Author : Luo Hao

Note : Since the project was launched to the 8w+ star, I have been receiving feedback that the basic part (the content of the first 15 days) is difficult for novices, and it is recommended to have a supporting video for explanation. Recently, I recreated a project called “Python-Core-50-Courses” from the basic part of the content . This part of the content was rewritten in a simpler and more popular way and accompanied by a video explanation . Beginners can pay attention to this new project. If domestic users are slow to access GitHub, they can also pay attention to the “Learn Python from scratch” column on my Zhihu Python-Jack . The column will continue to be updated, and the content of “data analysis” that everyone is looking forward to will also be coming soon Online, welcome everyone to pay attention to my columns, articles and answers in Zhihu .

The creation is not easy. Thank you for your rewards and support. The money will basically not be used to buy coffee. Instead, it will be donated to people in need through platforms such as Tencent Charity, Meituan Charity, and Didi Chuan ( click to learn about donations). Those who need to join the QQ exchange group can scan the QR code below. The exchange group will provide you with learning resources and answers to questions , and will continue to bring you free online Python experience courses and industry open courses , so stay tuned.

Analysis of Python application fields and employment situation

Simply put, Python is an “elegant”, “clear”, and “simple” programming language.

  • Low learning curve, even non-professionals can get started
  • Open source system, with a strong ecosystem
  • Interpreted language, perfect platform portability
  • Support object-oriented and functional programming
  • Ability to extend functions by calling C/C++ code
  • High degree of code standardization and strong readability

Python has its place in several popular fields.

  • Cloud Infrastructure-Python / Java / Go
  • DevOps-Python / Shell / Ruby / Go
  • Web crawler-Python / PHP / C++
  • Data analysis and mining-Python / R / Scala / Matlab
  • Machine Learning-Python / R / Java / Lisp

As a Python developer, the main areas of employment include:

  • Python server background development / game server development / data interface development engineer
  • Python Automation Operation and Maintenance Engineer
  • Python data analysis/ data visualization/ big data engineer
  • Python crawler engineer
  • Python chatbot development / image recognition and visual algorithms / deep learning engineer

The figure below shows the Python recruitment demand and salary ranking list in major cities (as of May 2018).

Python recruitment requirements and salary top 10

A few suggestions for beginners:

  • Make English as your working language. (Make English as your working language)
  • Practice makes perfect. (Practice makes perfect)
  • All experience comes from mistakes. (All experience comes from mistakes.)
  • Don’t be one of the leeches. (Don’t be one of the leeches.)
  • Either outstanding or out. (either outstanding or out)

Day01~15- Python language foundation

Day01-Getting to Know Python

  • Introduction to Python-History of Python / Advantages and Disadvantages of Python / Application Fields of Python
  • Set up a programming environment-Windows environment / Linux environment / MacOS environment
  • Run Python program from terminal-Hello, world / print function / run program
  • Use IDLE-Interactive Environment (REPL) / write multiple lines of code / run the program / exit IDLE
  • Comment-The role of comments / single-line comments / multi-line comments

Day02- Language Elements

  • Programs and bases-instructions and programs / Von Neumann machine / Binary and decimal / Octal and hexadecimal
  • Variables and types-variable naming / variable use / input function / check variable type / type conversion
  • Numbers and strings-integer / floating point / complex number / string / string basic operation / character encoding
  • Operators-Mathematical Operators / Assignment Operators / Comparison Operators / Logical Operators / Identity Operators / Priority of Operators
  • Application case-Convert Fahrenheit temperature to Celsius temperature / Enter the radius of the circle to calculate the circumference and area / Enter the year to determine whether it is a leap year

Day03- branch structure

  • Application scenarios of branch structure-condition / indentation / code block / flow chart
  • if statement-simple if / if-else structure / if-elif-else structure / nested if
  • Application case-User authentication / Interchange of imperial units and metric units / Decide what to do by rolling the dice / Converting percentile scores to grade system / Evaluation of piecewise function / Enter the length of three sides, if they can form a triangle, calculate the perimeter and area

Day04- Loop structure

  • Application scenarios of loop structure-condition/indentation/code block/flow chart
  • while loop-basic structure / break statement / continue statement
  • for loop-basic structure / range type / branch structure in loop / nested loop / end program early
  • Application case-1~100 summation / judging prime numbers / guessing the number game / printing the nine-nine table / printing the triangle pattern / monkeys eating peaches / hundred money and hundred chickens

Day05- Constructing program logic

  • Classic case: the number of daffodils / a hundred money and a hundred chickens / Craps gambling game
  • Practice questions: Fibonacci sequence / perfect numbers / prime numbers

Day06-Use of functions and modules

  • The role of the function-bad smell of the code / function module encapsulation
  • Define function-def statement/function name/parameter list/return statement/call custom function
  • Call function-Python built-in functions / import modules and functions
  • Function parameters-default parameters / variable parameters / keyword parameters / named keyword parameters
  • The return value of the function-no return value / return a single value / return multiple values
  • Scope issues-local scope / nested scope / global scope / built-in scope / scope-related keywords
  • Manage functions with modules-the concept of modules / use custom modules to manage functions / what happens when naming conflicts (the same module and different modules)

Day07- Strings and common data structures

  • The use of string-calculation length / subscript operation / slicing / common methods
  • Basic usage of list-define list / use the following table to access elements / subscript out of range / add element / delete element / modify element / slice / loop traversal
  • Common List Operations-Concatenation / Copy (Copy Elements and Copy Array) / Length / Sort / Reverse / Find
  • Generate list-use range to create a list of numbers / generate expressions / generators
  • Use of tuples-define tuples / use values ​​in tuples / modify tuple variables / tuples and list conversion
  • Basic usage of collection-the difference between collection and list / create collection / add element / delete element / empty
  • Common Set Operations-Intersection / Union / Difference / Symmetric Difference / Subset / Superset
  • The basic usage of the dictionary-the characteristics of the dictionary / create a dictionary / add elements / delete elements / take values ​​/ empty
  • Common dictionary operations-keys() method/ values() method/ items() method/ setdefault() method
  • Basic exercises-marquee effect / find the largest element in a list / statistical test score average / Fibonacci sequence / Yang Hui triangle
  • Comprehensive case-Two-color ball number selection / Tic-Tac-Toe

Day08- Fundamentals of Object - Oriented Programming

  • Classes and objects-what is a class / what is an object / other related object-oriented concepts
  • Define the class-basic structure / attributes and methods / constructor / destructor / str method
  • Use object-create object / send message to object
  • Four pillars of object-oriented-abstraction / encapsulation / inheritance / polymorphism
  • Basic exercises-define student class/ define clock class/ define graphics class/ define car class

Day09- Advanced Object Oriented

  • Attribute-class attribute / instance attribute / attribute accessor / attribute modifier / attribute deleter / use slots
  • Method in class-instance method/class method/static method
  • Operator overloading-add / sub / or /getitem / setitem / len / repr / gt / lt / le / ge / eq / ne / contains
  • The relationship between classes (of objects)-association/inheritance/dependency
  • Inheritance and polymorphism-what is the inheritance/ syntax of inheritance/ calling parent method/ method rewriting/ type determination/ multiple inheritance/ diamond inheritance (diamond inheritance) and C3 algorithm
  • Comprehensive Case-Salary Settlement System / Automatic Book Discount System / Customized Score

Day10- Graphical user interface and game development

  • Use tkinter to develop GUI programs
  • Use pygame third party library to develop game applications
  • “Big ball eats small ball” game

Day11- files and exceptions

  • Read file-read the entire file / line by line / file path
  • Write file-overwrite writing/additional writing/text file/binary file
  • Exception Handling-The importance of exception mechanism / try-except code block / else code block / finally code block / built-in exception type / exception stack / raise statement
  • Data Persistence-Overview of CSV File / Application of csv Module / JSON Data Format / Application of json Module

Day12- strings and regular expressions

  • Advanced string operations-escape characters / original strings / multi-line strings / in and not in operators / methods at the beginning of is / join and split methods / strip related methods / pyperclip module / invariant strings and variable characters Use of String/StringIO
  • Introduction to regular expressions-the role of regular expressions / metacharacters / escapes / quantifiers / grouping / zero-width assertions / greedy matching and lazy matching laziness / using the re module to implement regular expression operations (matching, searching, replacing, capturing)
  • Use regular expressions-re module / compile function / group and groups method / match method / search method / findall and finditer method / sub and subn method / split method
  • Application Case-Use regular expressions to verify the input string

Day13- Processes and threads

  • The concept of processes and threads-what is a process / what is a thread / multi-threaded application scenarios
  • Use process-fork function / multiprocessing module / process pool / inter-process communication
  • Use thread-thread module / threading module / Thread class / Lock class / Condition class / thread pool

Day14- Introduction to Network Programming and Network Application Development

  • Computer Network Foundation-History of Computer Network Development / “TCP-IP” Model / IP Address / Port / Protocol / Other Related Concepts
  • Network Application Mode-“Client-Server” Mode / “Browser-Server” Mode
  • Access network resources based on HTTP protocol-Network API overview / access URL / requests module / parse JSON format data
  • Python network programming-the concept of socket / socket module / socket function / create TCP server / create TCP client / create UDP server / create UDP client / SocketServer module
  • Email-SMTP protocol / POP3 protocol / IMAP protocol / smtplib module / poplib module / imaplib module
  • SMS Service-Call SMS Service Gateway

Day15- Image and document processing

  • Use Pillow to process pictures-picture reading and writing / picture synthesis / geometric transformation / color conversion / filter effect
  • Reading and writing Word documents-text content processing/paragraph/header and footer/style processing
  • Read and write Excel files-xlrd module / xlwt module
  • Generate PDF files-pypdf2 module / reportlab module

Day16~Day20- Advanced Python language

  • Common data structure
  • Advanced usage of functions-“first-class citizens” / higher-order functions / Lambda functions / scopes and closures / decorators
  • Advanced Object-Oriented Knowledge-“Three Pillars” / Relationship Between Classes / Garbage Collection / Magic Properties and Methods / Mixing / Metaclasses / Object-Oriented Design Principles / GoF Design Pattern
  • Iterators and generators-related magic methods / two ways to create generators /
  • Concurrent and asynchronous programming-multithreading/multiprocessing/asynchronous IO/async and await

Day21~30-Introduction to Web Front End

  • Use HTML tags to carry page content
  • Render the page with CSS
  • Handling interactive behaviors with JavaScript
  • jQuery entry and improvement
  • Getting started with Vue.js
  • Use of Element
  • Use of Bootstrap

Day31~35- Fun with Linux operating system

  • Operating system development history and Linux overview
  • Linux basic commands
  • Utilities in Linux
  • Linux file system
  • Vim editor application
  • Environment variables and Shell programming
  • Software installation and service configuration
  • Network access and management
  • Other related content

Day36~40- Database basics and advanced

  • Relational database MySQL
    • Overview of relational databases
    • MySQL installation and use
    • Use of SQL
      • DDL-Data Definition Language-create / drop / alter
      • DML-Data Manipulation Language-insert / delete / update / select
      • DCL-Data Control Language-grant / revoke
    • related information
      • Paradigm Theory-the guiding ideology for designing two-dimensional tables
      • Data integrity
      • Data consistency
    • Operate MySQL in Python
  • Getting started with NoSQL
    • NoSQL overview
    • Redis overview
    • Mongo overview

Day41~55- Practical Django

Day41- Django quick start

  • Web application working mechanism
  • HTTP request and response
  • Django framework overview
  • Quick start in 5 minutes

Day42- in - depth model

  • Relational database configuration
  • Use ORM to complete CRUD operations on the model
  • Use of management background
  • Django model best practices
  • Model definition reference

Day43- Static resources and Ajax requests

  • Load static resources
  • Ajax overview
  • Implement voting function with Ajax

Day44- Cookie and Session

  • Implement user tracking
  • The relationship between cookie and session
  • Django framework support for session
  • Cookie read and write operations in view functions

Day45- reports and logs

  • Modify the response header through HttpResponse
  • Use StreamingHttpResponse to process large files
  • Use xlwt to generate Excel reports
  • Use reportlab to generate PDF reports
  • Use ECharts to generate front-end charts

Day46- Log and debug toolbar

  • Configuration log
  • Configure Django-Debug-Toolbar
  • Optimize ORM code

Day47- Middleware Application

  • What is middleware
  • Middleware built into Django framework
  • Custom middleware and its application scenarios

Day48- Introduction to Front-end and Back- end Separation

  • Return data in JSON format
  • Render the page with Vue.js

Day49- Introduction to RESTful architecture and DRF

Day50- RESTful architecture and advanced DRF

Day51- Use cache

  • The first law of website optimization
  • Use Redis to provide caching service in Django project
  • Read and write cache in view function
  • Use decorators to implement page caching
  • Provide caching service for data interface

Day52- file upload

  • File upload form control and image file preview
  • How the server handles uploaded files

Day53- Asynchronous tasks and timing tasks

  • The second law of website optimization
  • Configure message queue service
  • Use celery to implement task asynchronization in the project
  • Use celery to implement timing tasks in the project

Day54- unit testing

Day55- project online

  • Unit testing in Python
  • Django framework’s support for unit testing
  • Use version control system
  • Configure and use uWSGI
  • Dynamic separation and Nginx configuration
  • Configure HTTPS
  • Configure domain name resolution

Day56~60- Actual Combat Flask

Day56- Getting started with Flask

Day57- Use of templates

Day58- Form processing

Day59- database operations

Day60- actual project

Day61~65- Actual Tornado

Day61- Preliminary Knowledge

  • Concurrent programming
  • I/O mode and event driven

Day62- Getting started with Tornado

  • Tornado overview
  • 5 minutes to get started with Tornado
  • Route resolution
  • Request handler

Day63- Asynchronization

  • The use of aiomysql and aioredis

Day64- Application of WebSocket

  • Introduction to WebSocket
  • WebSocket server-side programming
  • WebSocket client programming
  • Project: Web chat room

Day65- Project Actual Combat

  • Front-end and back-end separate development and interface document writing
  • Use Vue.js to achieve front-end rendering
  • Use ECharts to realize report function
  • Use WebSocket to implement push service

Day66~75- crawler development

Day66- Web crawlers and related tools

  • The concept of web crawler and its application field
  • Discussion on the legality of web crawlers
  • Tools for developing web crawlers
  • The composition of a crawler

Day67- Data Collection and Analysis

  • Standards and tripartite libraries for data collection
  • Three ways of page parsing: regular expression parsing/XPath parsing/CSS selector parsing

Day68- store data

  • How to store massive amounts of data
  • Implement data caching

Day69- concurrent download

  • Multithreading and multiprocessing
  • Asynchronous I/O and coroutines
  • The use of async and await keywords
  • Application of the three party library aiohttp

Day70- Parse dynamic content

  • JavaScript reverse engineering
  • Use Selenium to get dynamic content

Day71- Form interaction and verification code processing

  • Auto submit form
  • Cookie pool application
  • Verification code processing

Day72- Getting Started with Scrapy

  • Scrapy crawler framework overview
  • Install and use Scrapy

Day73- Scrapy Advanced Application

  • Spider usage
  • Middleware application: download middleware / spider middleware
  • Scrapy docks with Selenium to capture dynamic content
  • Scrapy deployed to Docker

Day74- Scrapy distributed implementation

  • Principles of distributed crawlers
  • Scrapy distributed implementation
  • Use Scrapyd to implement distributed deployment

Day75- Reptile Project Actual Combat

  • Crawling recruitment website data
  • Crawling real estate industry data
  • Crawling used car trading platform data

Day76~90- Data Analysis and Machine Learning

Reminder : The content of data analysis and machine learning is in the code folder. It is the code and notes written in jupyter notebook. You need to start jupyter notebook and then open the corresponding file for learning. Relevant documents will continue to be supplemented in 2020, and I hope you will continue to pay attention.

Day76- Machine Learning Fundamentals

Day77- Application of Pandas

Day78-Application of NumPy and SciPy

Day79- Matplotlib and data visualization

Day80- k nearest neighbor (KNN) classification

Day81- Decision Tree

Day82- Bayesian Classification

Day83- Support Vector Machine (SVM)

Day84- K-means clustering

Day85- regression analysis

Day86- Introduction to Big Data Analysis

Day87- Advanced Big Data Analysis

Day88- Getting Started with Tensorflow

Day89- Tensorflow combat

Day90- Recommended system combat

Day91~100- Team project development

Day 91: Problems and solutions for team project development

  1. Software process model
  • Classic process model (waterfall model)
    • Feasibility analysis (whether the research is done or not), and output the “feasibility analysis report”.
    • Requirement analysis (study what to do), output “requirements specification” and product interface prototype diagram.
    • Outline design and detailed design, output conceptual model diagram (ER diagram), physical model diagram, class diagram, sequence diagram, etc.
    • Coding/testing.
    • Go online/maintenance.The biggest disadvantage of the waterfall model is that it cannot embrace changes in requirements. The product can only be seen after the entire process is over, and the team morale is low.
  • Agile Development (Scrum)-Product Owner, Scrum Master, R&D Staff-Sprint
    • Product Backlog (user story, product prototype).
    • Planning meetings (evaluation and budgeting).
    • Daily development (stand-up meeting, Pomodoro Technique, pair programming, test first, code refactoring…).
    • Fix bugs (problem description, reproduction steps, testers, designees).
    • release version.
    • Review meeting (Showcase, users need to participate).
    • Retrospective meeting (a summary of the current iteration cycle).

Supplement: Manifesto for Agile Software Development

* **Individuals and interactions are**  higher than processes and tools
* **Working software is**  higher than detailed documentation
* **Customer cooperation is**  higher than contract negotiation
* **Responding to changes is**  better than following a plan

Roles: product owner (the person who decides what to do and can approve the demand), team leader (solve various problems, focus on how to work better, shield the external influence on the development team), development team (project executive, Specifically refers to developers and testers).

Preparation work: business case and capital, contract, vision, initial product requirements, initial release plan, shareholding, and team formation.

Agile teams usually have 8-10 people.

Estimation of workload: Quantify the development tasks, including prototype, logo design, UI design, front-end development, etc., try to break down each work to the minimum amount of tasks, the minimum task amount standard is that the working time cannot exceed two days, and then estimate the overall project time . Post each task on the Kanban board, which is divided into three parts: to do (to be completed), in progress (in progress) and done (completed).

  1. Project team formation
  • Team composition and roles

Note: Thank you Ms. Fu Xiangying for helping me draw the following beautiful company organization chart.

  • Programming specification and code review (flake8, pylint)
  • Some “conventions” in Python (please refer to “Python Conventions-How to Write Pythonic Code” )
  • Reasons affecting code readability:
    • Code comments are too few or no comments
    • Code breaks language best practices
    • Anti-pattern programming (spaghetti code, copy-paste programming, conceit programming,…)
  1. Introduction to team development tools
Project topic selection and business understanding
  1. Topic selection range setting
  • CMS (user terminal): news aggregation website, Q&A/sharing community, film review/book review website, etc.
  • MIS (user side + management side): KMS, KPI assessment system, HRS, CRM system, supply chain system, warehouse management system, etc.
  • App background (management terminal + data interface): second-hand trading, newspapers and magazines, niche e-commerce, news information, travel, social, reading, etc.
  • Other types: own industry background and work experience, business is easy to understand and control.
  1. Requirements understanding, module division and task allocation
  • Need understanding: brainstorming and competitive product analysis.
  • Module division: draw a mind map (XMind), each module is a branch node, and each specific function is a leaf node (expressed in verbs). It is necessary to ensure that each leaf node cannot regenerate a new node, and determine each leaf The importance, priority and workload of the node.
  • Task assignment: The project leader assigns tasks to each team member according to the above indicators.
  1. Develop project schedule (updated daily)
Module Features personnel status carry out Working hours Plan started Actual start End of plan Actual end Remarks
comment add comment Wang Dahui Ongoing 50% 4 2018/8/7 2018/8/7
Delete comment Wang Dahui wait 0% 2 2018/8/7 2018/8/7
View comments Bai Yuanfang Ongoing 20% 4 2018/8/7 2018/8/7 Code review required
Comment vote Bai Yuanfang wait 0% 4 2018/8/8 2018/8/8
  1. OOAD and database design
  • UML (Unified Modeling Language) class diagramuml
  • Create table from model (forward engineering)

python makemigrations app python migrate

  • Use PowerDesigner to draw physical model diagrams.
  • Create model from data table (reverse engineering)

python inspectdb > app/

Day 92: Docker container explained

  1. Introduction to Docker
  2. Install Docker
  3. Use Docker to create containers (Nginx, MySQL, Redis, Gitlab, Jenkins)
  4. Build Docker image (write Dockerfile and related instructions)
  5. Container Orchestration (Docker-compose)
  6. Cluster Management (Kubernetes)

Day 93: MySQL performance optimization

Day 94: Web API interface design

Day 95: [Develop commercial projects using Django](./Day91-100/95. Use Django to develop commercial

Public issues in project development
  1. Database configuration (multi-database, master-slave replication, database routing)
  2. Cache configuration (partition cache, key settings, timeout settings, master-slave replication, failure recovery (sentinel))
  3. Log configuration
  4. Analysis and debugging (Django-Debug-ToolBar)
  5. Useful Python modules (date calculation, image processing, data encryption, three-party API)
REST API design
  1. RESTful architecture
  1. API interface document writing
  1. django-REST-framework application
Analysis of key and difficult points in the project
  1. Use cache to relieve database pressure-Redis
  2. Use message queues for decoupling and peak clipping-Celery + RabbitMQ

Day 96: Software testing and automated testing

unit test
  1. Type of test
  2. Write unit tests (unittest, pytest, nose2, tox, ddt,…)
  3. Test coverage (coverage)
Project deployment
  1. Preparation before deployment
  • Key settings (SECRET_KEY / DEBUG / ALLOWED_HOSTS / cache / database)
  • Log related configuration
  1. Review of Linux commonly used commands
  2. Installation and configuration of Linux common services
  3. Use of uWSGI/Gunicorn and Nginx
  • Comparison of Gunicorn and uWSGI
    • For simple applications that do not require a lot of customization, Gunicorn is a good choice. The learning curve of uWSGI is much steeper than that of Gunicorn. Gunicorn’s default parameters can already be adapted to most applications.
    • uWSGI supports heterogeneous deployment.
    • Since Nginx itself supports uWSGI, Nginx and uWSGI are generally bundled and deployed online, and uWSGI is a fully functional and highly customized WSGI middleware.
    • In terms of performance, Gunicorn and uWSGI are actually comparable.
  1. Use virtualization technology (Docker) to deploy test and production environments
Performance Testing
  1. Use of AB
  2. Use of SQLslap
  3. Use of sysbench
automated test
  1. Use Shell and Python for automated testing
  2. Use Selenium to implement automated testing
  • Selenium IDE
  • Selenium WebDriver
  • Selenium Remote Control
  1. Introduction of testing tool Robot Framework

Day 97: Analysis of technical points of e-commerce website

Day 98: Project deployment and performance tuning

  1. MySQL database tuning
  2. Web server performance optimization
  • Nginx load balancing configuration
  • Keepalived achieves high availability
  1. Code performance tuning
  • Multithreading
  • Asynchronous
  1. Static resource access optimization
  • Cloud storage
  • CDN

Day 99: Public questions in the interview

Day 100: Python interview question set


pls put it in english

1 Like

Use Google Translate.


Nice share…thanks a lot.

thanks for sharing, i really want to learn programming

Can’t appreciate enouh… :slight_smile:

Complete guide in itself

Good, thanks.