BogoToBogo
  • Home
  • About
  • Big Data
  • Machine Learning
  • AngularJS
  • Python
  • C++
  • go
  • DevOps
  • Kubernetes
  • Algorithms
  • More...
    • Qt 5
    • Linux
    • FFmpeg
    • Matlab
    • Django 1.8
    • Ruby On Rails
    • HTML5 & CSS

Python Introduction 2020

python_logo




Bookmark and Share





bogotobogo.com site search:

What Can We Do with Python?

Component Integration

Python scripts can easily communicate with other application variety of integration mechanisms.


Python code can invoke C/C++ libraries and can be called from C/C++.
For example, integrating a C library into Python enables Python to test and launch the library's component, and embedding Python in a product enables onsite customizations to be coded without having to recompile the entire product.

It can integrate with Java and .Net components, and it can communicate over frameworks such as COM. It also can interface with devices over serial ports, and can interact over networks with interfaces like SOAP, XML-RPC, and COBRA.

Another alternative ways to script components are:

  1. Cython system allows coders to mix Python and C-code.
  2. SWIG and SIP code generators can automate much of the work needed to link compiled components into Python.
  3. Python's COM support.
  4. Iron Python .NET frameworks based implementation.
  5. Jython: Java-based implementation.
  6. COBRA toolkits for Python.


Database Programming

For general database, Python gives us interfaces to all RDBS - Sybase, Oracle, MySQL, PostgreSQL, and SQLite.
Python's standard pickle module provides a simple object persistence system.



GUI

Python's simplicity and rapid turnaround make it a good match for graphical user interface programming. Python comes with a object-oriented interface to the Tk GUI API called tkinter which allows Python to implement portable GUIs with native look and feel.

Python/tkinter GUIs run unchanged for different platforms.



Internet Scripting

Python scripts can:

  1. Communicate over sockets.
  2. Extract form information sent to server-side CGI scripts.
  3. Transfer files by FTP.
  4. Parse, generate, and analyze XML files.
  5. Send, receive, compose, and parse email.
  6. Fetch web pages by URLs.
  7. Parse the HTML and XML of fetched web pages.
  8. Communicate over XML-RPC, SOAP, and Telnet.

There are framework web development packages for Python:

  1. Django
  2. TurboGears
  3. web2py
  4. Pylons
  5. Zope
  6. WebWare


System Programming

Python's built-in interfaces to OS services make it ideal for writing portable shell tools.
Python's standard library comes with POSIX binding and support for all the usual OS tools.





jirisanWinter




Running Python Programs

Python Interpreter

The Python code we write must always be run by the Interpreter.
To enable it, we must install a Python interpreter on our machine.
When the Python package is installed on our machine, it generates number of components:

  1. Interpreter
  2. Library
generated_files


Python Execution

Make a file hello.py with the following line.

print("Hello world!")

The extension py is not required but for consistency, Python file usually has that extension.

If we run the file on a command prompt window:

C:\temp > python hello.py
Hello World

We typed the code into text file, and we run the file through the interpreter. Simple!
But let's think about the runtime structure of Python.

When we instruct Python to run our script, there are a few steps that Python carries out before our code actually starts crunching away:

  1. It is compiled to bytecode.
  2. Then it is routed to virtual machine.



Python Byte Code

When we execute a source code, Python compiles it into a byte code. Compilation is a translation step, and the byte code is a low-level platform-independent representation of source code. Note that the Python byte code is not binary machine code (e.g., instructions for an Intel chip).

Actually, Python translate each statement of the source code into byte code instructions by decomposing them into individual steps. The byte code translation is performed to speed execution.
Byte code can be run much more quickly than the original source code statements. It has .pyc extension and it will be written if it can write to our machine.

So, next time we run the same program, Python will load the .pyc file and skip the compilation step unless it's been changed. Python automatically checks the timestamps of source and byte code files to know when it must recompile. If we resave the source code, byte code is automatically created again the next time the program is run.

If Python cannot write the byte code files to our machine, our program still works. The byte code is generated in memory and simply discarded on program exit. But because .pyc files speed startup time, we may want to make sure it has been written for larger programs.

Let's summarize what happens behind the scenes.
When a Python executes a program, Python reads the .py into memory, and parses it in order to get a bytecode, then goes on to execute. For each module that is imported by the program, Python first checks to see whether there is a precompiled bytecode version, in a .pyo or .pyc, that has a timestamp which corresponds to its .py file. Python uses the bytecode version if any. Otherwise, it parses the module's .py file, saves it into a .pyc file, and uses the bytecode it just created.

Byte code files are also one way of shipping Python codes. Python will still run a program if all it can find are .pyc files, even if the original .py source files are not there.




Python Virtual Machine (PVM)

Once our program has been compiled into byte code, it is shipped off for execution to Python Virtual Machine (PVM). The PVM is not a separate program. It need not be installed by itself. Actually, the PVM is just a big loop that iterates through our byte code instruction, one by one, to carry out their operations. The PVM is the runtime engine of Python. It's always present as part of the Python system. It's the component that truly runs our scripts. Technically it's just the last step of what is called the Python interpreter.




Python Performance

Unlike other compiled languages, Python code runs immediately after it is written. Byte code is a Python-specific representation. Since byte code is not a binary machine instructions, it requires more work than CPU instructions.

On the other hand, unlike other interpreters, there is still an internal compile step - Python does not need to reanalyze and reparse each source statement repeatedly. So, Python code runs at speeds somewhere between those of a traditional compiled language and a classic interpreted language.




Python Development

There is no distinction between the development and execution environments. In other words, the compiler is always right there at runtime and is part of the system that runs the program. All we have in Python is runtime. There is no initial compile-time phase at all and everything happens as the program is running.

This even includes operation such as the creation of functions and classes and the linking of modules. Such events occur before execution in more static languages, but in Python, they happen as programs execute.

This adds a more dynamic flavor to the Python - it is often very convenient for Python programs to construct and execute other Python programs at runtime. The eval and exec built-ins, for example, accept and run strings containing Python program code.

This structure is why Python lends itself to customization. Because Python code can be changed on the fly, users can modify the Python parts of a system onsite without needing to compile the entire system's code.




Frozen Binaries

It is possible to turn our Python programs into true executables. This is known as frozen binaries. Frozen binaries bundles together the byte code of our program files, along with the PVM interpreter and any Python support files our program needs, into a single package, a single binary executable program like .exe file on Windows. In other words, the byte code and PVM are merged into a single component.

Frozen binaries are not the same as the output of a true compiler. They run byte code through a virtual machine. Frozen binaries are not small because they contain PVM, but they are not unusually large either.

Three primary systems are capable of generating frozen binaries:

  1. py2exe for Windows.
  2. PyInstaller is similar to py2exe but also works on Linux and Unix.
  3. freeze.





JooWangMt












Ph.D. / Golden Gate Ave, San Francisco / Seoul National Univ / Carnegie Mellon / UC Berkeley / DevOps / Deep Learning / Visualization

YouTubeMy YouTube channel

Sponsor Open Source development activities and free contents for everyone.

Thank you.

- K Hong







Python tutorial



Python Home

Introduction

Running Python Programs (os, sys, import)

Modules and IDLE (Import, Reload, exec)

Object Types - Numbers, Strings, and None

Strings - Escape Sequence, Raw String, and Slicing

Strings - Methods

Formatting Strings - expressions and method calls

Files and os.path

Traversing directories recursively

Subprocess Module

Regular Expressions with Python

Regular Expressions Cheat Sheet

Object Types - Lists

Object Types - Dictionaries and Tuples

Functions def, *args, **kargs

Functions lambda

Built-in Functions

map, filter, and reduce

Decorators

List Comprehension

Sets (union/intersection) and itertools - Jaccard coefficient and shingling to check plagiarism

Hashing (Hash tables and hashlib)

Dictionary Comprehension with zip

The yield keyword

Generator Functions and Expressions

generator.send() method

Iterators

Classes and Instances (__init__, __call__, etc.)

if__name__ == '__main__'

argparse

Exceptions

@static method vs class method

Private attributes and private methods

bits, bytes, bitstring, and constBitStream

json.dump(s) and json.load(s)

Python Object Serialization - pickle and json

Python Object Serialization - yaml and json

Priority queue and heap queue data structure

Graph data structure

Dijkstra's shortest path algorithm

Prim's spanning tree algorithm

Closure

Functional programming in Python

Remote running a local file using ssh

SQLite 3 - A. Connecting to DB, create/drop table, and insert data into a table

SQLite 3 - B. Selecting, updating and deleting data

MongoDB with PyMongo I - Installing MongoDB ...

Python HTTP Web Services - urllib, httplib2

Web scraping with Selenium for checking domain availability

REST API : Http Requests for Humans with Flask

Blog app with Tornado

Multithreading ...

Python Network Programming I - Basic Server / Client : A Basics

Python Network Programming I - Basic Server / Client : B File Transfer

Python Network Programming II - Chat Server / Client

Python Network Programming III - Echo Server using socketserver network framework

Python Network Programming IV - Asynchronous Request Handling : ThreadingMixIn and ForkingMixIn

Python Coding Questions I

Python Coding Questions II

Python Coding Questions III

Python Coding Questions IV

Python Coding Questions V

Python Coding Questions VI

Python Coding Questions VII

Python Coding Questions VIII

Python Coding Questions IX

Python Coding Questions X

Image processing with Python image library Pillow

Python and C++ with SIP

PyDev with Eclipse

Matplotlib

Redis with Python

NumPy array basics A

NumPy Matrix and Linear Algebra

Pandas with NumPy and Matplotlib

Celluar Automata

Batch gradient descent algorithm

Longest Common Substring Algorithm

Python Unit Test - TDD using unittest.TestCase class

Simple tool - Google page ranking by keywords

Google App Hello World

Google App webapp2 and WSGI

Uploading Google App Hello World

Python 2 vs Python 3

virtualenv and virtualenvwrapper

Uploading a big file to AWS S3 using boto module

Scheduled stopping and starting an AWS instance

Cloudera CDH5 - Scheduled stopping and starting services

Removing Cloud Files - Rackspace API with curl and subprocess

Checking if a process is running/hanging and stop/run a scheduled task on Windows

Apache Spark 1.3 with PySpark (Spark Python API) Shell

Apache Spark 1.2 Streaming

bottle 0.12.7 - Fast and simple WSGI-micro framework for small web-applications ...

Flask app with Apache WSGI on Ubuntu14/CentOS7 ...

Selenium WebDriver

Fabric - streamlining the use of SSH for application deployment

Ansible Quick Preview - Setting up web servers with Nginx, configure enviroments, and deploy an App

Neural Networks with backpropagation for XOR using one hidden layer

NLP - NLTK (Natural Language Toolkit) ...

RabbitMQ(Message broker server) and Celery(Task queue) ...

OpenCV3 and Matplotlib ...

Simple tool - Concatenating slides using FFmpeg ...

iPython - Signal Processing with NumPy

iPython and Jupyter - Install Jupyter, iPython Notebook, drawing with Matplotlib, and publishing it to Github

iPython and Jupyter Notebook with Embedded D3.js

Downloading YouTube videos using youtube-dl embedded with Python

Machine Learning : scikit-learn ...

Django 1.6/1.8 Web Framework ...


Sponsor Open Source development activities and free contents for everyone.

Thank you.

- K Hong






OpenCV 3 image and video processing with Python



OpenCV 3 with Python

Image - OpenCV BGR : Matplotlib RGB

Basic image operations - pixel access

iPython - Signal Processing with NumPy

Signal Processing with NumPy I - FFT and DFT for sine, square waves, unitpulse, and random signal

Signal Processing with NumPy II - Image Fourier Transform : FFT & DFT

Inverse Fourier Transform of an Image with low pass filter: cv2.idft()

Image Histogram

Video Capture and Switching colorspaces - RGB / HSV

Adaptive Thresholding - Otsu's clustering-based image thresholding

Edge Detection - Sobel and Laplacian Kernels

Canny Edge Detection

Hough Transform - Circles

Watershed Algorithm : Marker-based Segmentation I

Watershed Algorithm : Marker-based Segmentation II

Image noise reduction : Non-local Means denoising algorithm

Image object detection : Face detection using Haar Cascade Classifiers

Image segmentation - Foreground extraction Grabcut algorithm based on graph cuts

Image Reconstruction - Inpainting (Interpolation) - Fast Marching Methods

Video : Mean shift object tracking

Machine Learning : Clustering - K-Means clustering I

Machine Learning : Clustering - K-Means clustering II

Machine Learning : Classification - k-nearest neighbors (k-NN) algorithm




Machine Learning with scikit-learn



scikit-learn installation

scikit-learn : Features and feature extraction - iris dataset

scikit-learn : Machine Learning Quick Preview

scikit-learn : Data Preprocessing I - Missing / Categorical data

scikit-learn : Data Preprocessing II - Partitioning a dataset / Feature scaling / Feature Selection / Regularization

scikit-learn : Data Preprocessing III - Dimensionality reduction vis Sequential feature selection / Assessing feature importance via random forests

Data Compression via Dimensionality Reduction I - Principal component analysis (PCA)

scikit-learn : Data Compression via Dimensionality Reduction II - Linear Discriminant Analysis (LDA)

scikit-learn : Data Compression via Dimensionality Reduction III - Nonlinear mappings via kernel principal component (KPCA) analysis

scikit-learn : Logistic Regression, Overfitting & regularization

scikit-learn : Supervised Learning & Unsupervised Learning - e.g. Unsupervised PCA dimensionality reduction with iris dataset

scikit-learn : Unsupervised_Learning - KMeans clustering with iris dataset

scikit-learn : Linearly Separable Data - Linear Model & (Gaussian) radial basis function kernel (RBF kernel)

scikit-learn : Decision Tree Learning I - Entropy, Gini, and Information Gain

scikit-learn : Decision Tree Learning II - Constructing the Decision Tree

scikit-learn : Random Decision Forests Classification

scikit-learn : Support Vector Machines (SVM)

scikit-learn : Support Vector Machines (SVM) II

Flask with Embedded Machine Learning I : Serializing with pickle and DB setup

Flask with Embedded Machine Learning II : Basic Flask App

Flask with Embedded Machine Learning III : Embedding Classifier

Flask with Embedded Machine Learning IV : Deploy

Flask with Embedded Machine Learning V : Updating the classifier

scikit-learn : Sample of a spam comment filter using SVM - classifying a good one or a bad one




Machine learning algorithms and concepts

Batch gradient descent algorithm

Single Layer Neural Network - Perceptron model on the Iris dataset using Heaviside step activation function

Batch gradient descent versus stochastic gradient descent

Single Layer Neural Network - Adaptive Linear Neuron using linear (identity) activation function with batch gradient descent method

Single Layer Neural Network : Adaptive Linear Neuron using linear (identity) activation function with stochastic gradient descent (SGD)

Logistic Regression

VC (Vapnik-Chervonenkis) Dimension and Shatter

Bias-variance tradeoff

Maximum Likelihood Estimation (MLE)

Neural Networks with backpropagation for XOR using one hidden layer

minHash

tf-idf weight

Natural Language Processing (NLP): Sentiment Analysis I (IMDb & bag-of-words)

Natural Language Processing (NLP): Sentiment Analysis II (tokenization, stemming, and stop words)

Natural Language Processing (NLP): Sentiment Analysis III (training & cross validation)

Natural Language Processing (NLP): Sentiment Analysis IV (out-of-core)

Locality-Sensitive Hashing (LSH) using Cosine Distance (Cosine Similarity)




Artificial Neural Networks (ANN)

[Note] Sources are available at Github - Jupyter notebook files

1. Introduction

2. Forward Propagation

3. Gradient Descent

4. Backpropagation of Errors

5. Checking gradient

6. Training via BFGS

7. Overfitting & Regularization

8. Deep Learning I : Image Recognition (Image uploading)

9. Deep Learning II : Image Recognition (Image classification)

10 - Deep Learning III : Deep Learning III : Theano, TensorFlow, and Keras









Contact

BogoToBogo
contactus@bogotobogo.com

Follow Bogotobogo

About Us

contactus@bogotobogo.com

YouTubeMy YouTube channel
Pacific Ave, San Francisco, CA 94115

Pacific Ave, San Francisco, CA 94115

Copyright © 2024, bogotobogo
Design: Web Master