BogoToBogo
  • Home
  • About
  • Big Data
  • Machine Learning
  • AngularJS
  • Python
  • C++
  • go
  • DevOps
  • Kubernetes
  • Algorithms
  • More...
    • Qt 5
    • Linux
    • FFmpeg
    • Matlab
    • Django 1.8
    • Ruby On Rails
    • HTML5 & CSS

Coding Questions - 2024

python_logo




Bookmark and Share





bogotobogo.com site search:

Merging two sorted list

We have two sorted lists, and we want to write a function to merge the two lists into one sorted list:

a = [3, 4, 6, 10, 11, 18]
b = [1, 5, 7, 12, 13, 19, 21]

Here is our code:

a = [3, 4, 6, 10, 11, 18]
b = [1, 5, 7, 12, 13, 19, 21]
c = []

while a and b:
    if a[0] < b[0]:
        c.append(a.pop(0))
    else:
        c.append(b.pop(0))
        
# either a or b could still be non-empty
print c + a + b

The output:

[1, 3, 4, 5, 6, 7, 10, 11, 12, 13, 18, 19, 21]

A little bit more compact version using list.extend():

a = [3, 4, 6, 10, 11, 18]
b = [1, 5, 7, 12, 13, 19, 21]

a.extend(b)
c = sorted(a)
print c

Note that the list.extend() is different from list.append():

[1, 3, 5, 7, 9, 11, [2, 4, 6]]  # a.append(b)
[1, 3, 5, 7, 9, 11, 2, 4, 6]    # a.extend(b)




Get word frequency - initializing dictionary

We'll see two ways of initializing dictionary by solving word frequency problem.

#!/usr/bin/python
ss = """Nory was a Catholic because her mother was a Catholic, 
and Nory's mother was a Catholic because her father was a Catholic, 
and her father was a Catholic because his mother was a Catholic, 
or had been."""

words = ss.split()
d = {}.fromkeys(words,0)

# or we can use this to initialize a dict
d = {x:0 for x in words}

for w in words:
    d[w] += 1
print d

ds = sorted(d.items(), key=lambda x:x[1], reverse=True)
print(ds)

# or we can use this for the key
import operator
ds2 = sorted(d.items(), key=operator.itemgetter(1), reverse=True)
print(ds2)

We initialized the dictionary with 0 using fromkeys(), and the output should look like this:

{'a': 6, 'Catholic': 3, 'and': 2, 'because': 3, 'her': 3, 'had': 1, 'Nory': 1, 'father': 2, 
"Nory's": 1, 'his': 1, 'been.': 1, 'mother': 3, 'was': 6, 'or': 1, 'Catholic,': 3}

Here is another way of initializing a dictionary:

d = {}
for w in ss.split():
    d[w] = d.get(w,0) + 1
print d

The third way of dictionary initialization using collections.defaultdict(int) which is convenient for counting:

from collections import defaultdict
d = defaultdict(int)
for w in words:
    d[w] += 1
print d




Initializing dictionary with list - I

Sometimes we may want to construct dictionary whose values are lists.

In the following example, we make a dictionary like {'Country': [cities,...], }:

cities = {'San Francisco': 'US', 'London':'UK',
        'Manchester':'UK', 'Paris':'France',
        'Los Angeles':'US', 'Seoul':'Korea'}

# => {'US':['San Francisco', 'Los Angeles'], 'UK':[,], ...}

from collections import defaultdict
# using collections.defaultdict()
d1 = defaultdict(list) # initialize dict with list
for k,v in cities.items():
    d1[v].append(k)
print(f'd1 = {d1}')

# using dict.setdefault(key, default=None)
d2 = {}
for k,v in cities.items():
       d2.setdefault(v,[]).append(k)
print(f'd2 = {d2}')

Output:

d1 = defaultdict(<type 'list'>, {'US': ['San Francisco', 'Los Angeles'], 'UK': ['London', 'Manchester'], 'France': ['Paris'], 'Korea': ['Seoul']})
d2 = {'US': ['San Francisco', 'Los Angeles'], 'UK': ['London', 'Manchester'], 'France': ['Paris'], 'Korea': ['Seoul']}

Or we can use this code:

#! /usr/bin/python

cities = {'San Francisco': 'US', 'London':'UK',
        'Manchester':'UK', 'Paris':'France',
        'Los Angeles':'US', 'Seoul':'Korea'}

# => {'US':['San Francisco', 'Los Angeles'], 'UK':[,], ...}

a = [v for k,v in cities.items()]
print(a)

ds = {x:[] for x in set(a)}
print(ds)

for k,v in cities.items():
	ds[v].append(k)

print(ds)    

Output:

['UK', 'Korea', 'France', 'UK', 'US', 'US']
{'Korea': [], 'UK': [], 'US': [], 'France': []}
{'Korea': ['Seoul'], 'UK': ['Manchester', 'London'], 'US': ['Los Angeles', 'San Francisco'], 'France': ['Paris']}    




Initializing dictionary with list - II

A little bit simpler problem. We have a list of numbers:

L = [1,2,4,8,16,32,64,128,256,512,1024,32768,65536,4294967296]

We want to make a dictionary with the number of digits as the key and list of numbers the value:

 {1: [1, 2, 4, 8], 2: [16, 32, 64], 3: [128, 256, 512], 4: [1024], 5: [32768, 65536], 10: [4294967296]})

The code looks like this:

L = [1,2,4,8,16,32,64,128,256,512,1024,32768,65536,4294967296]

from collections import defaultdict
d = defaultdict(list)
for i in L:
    d[len(str(i))].append(i)
print d
print {k:v for k,v in d.items()}

Output:

defaultdict(<type 'list'>, {1: [1, 2, 4, 8], 2: [16, 32, 64], 3: [128, 256, 512], 4: [1024], 5: [32768, 65536], 10: [4294967296]})
{1: [1, 2, 4, 8], 2: [16, 32, 64], 3: [128, 256, 512], 4: [1024], 5: [32768, 65536], 10: [4294967296]}

More examples: Word Frequency





map, filter, and reduce

Using map, filter, reduce, write a code that create a list of (n)**2 for range(10) for even integers:

l = [x for x in range(10) if x % 2 == 0]
print l

m = filter(lambda x:x % 2 == 0, [x for x in range(10)] )
# m = list( filter(lambda x:x % 2 == 0, [x for x in range(10)] ) ) # python3
print m

o = map(lambda x: x**2, m)
# o = list( map(lambda x: x**2, m) ) # python3
print o


p = reduce(lambda x,y:x+y, o)
# import functools . # python3
# p = functools.reduce(lambda x,y:x+y, o) # python3
print p

Output:

[0, 2, 4, 6, 8]
[0, 2, 4, 6, 8]
[0, 4, 16, 36, 64]
120



Write a function f()

Q: We have the following code with unknown function f(). In f(), we do not want to use return, instead, we may want to use generator.

for x in f(5):
    print x,

The output looks like this:

0 1 8 27 64

Write a function f() so that we can have the output above.

We may use the following f() to get the same output:

def f(n):
   return [x**3 for x in range(n)]

But we want to use generator not using return.

So, the answer should look like this:

def f(n):
    for x in range(n):
        yield x**3

The yield enables a function to comeback where it left off when it is called again. This is the critical difference from a regular function. A regular function cannot comes back where it left off. The yield keyword helps a function to remember its state.

A generator function is a way to create an iterator. A new generator object is created and returned each time we call a generator function. A generator yields the values one at a time, which requires less memory and allows the caller to get started processing the first few values immediately.

Another example of using yield:

Let's build the primes() function so that I fills the n one at a time, and comes back to primes() function until n > 100.

def isPrime(n):
   if n == 1:
      return False
   for t in range(2,n):
      if n % t == 0:
         return False
   return True

for n in primes():
   print n,

The print out from the for-loop should look like this:

2 3 5 7 11 ... 83 89 97

Here is the primes() function:

def primes(n=1):
   while n < 100:
      # yields n instead of returns n
      if isPrime(n): yield n
      # next call it will increment n by 1
      n += 1



Here is a more practical sample of code which I used for Natural Language Processing(NLP).

Suppose we have a huge data file that has hundred millions of lines. So, it may well exceed our computer's memory. In this case, we may want to take so called out-of-core approach: we process data in batch (partially, one by one) rather than process it at once. This saves us from the memory issue when we deal with big data set.

So, we want to use yield command. In the following sample, we do process three lines at a time.

Here is the code:

def stream_docs(path):
    with open(path, 'rb') as lines:
        for line in lines:
            text, label = line[:-3], line[-3:-1]
            yield text, label
            
def get_minibatch(doc_stream, size):
    docs, y = [], []
    try:
        for _ in range(size):
            text, label = next(doc_stream)
            docs.append(text)
            y.append(label)
    except StopIteration:
        return None, None
    return docs, y

doc_stream = stream_docs(path='./test.txt')

for _ in range(100):
    X_train, y_train = get_minibatch(doc_stream, size=3)
    if not X_train:
        break
    print 'X_train, y_train=', X_train, y_train

The yield makes the stream_docs() to return a generator which is always an iterator:

generators-iterators-iterables.png

(note) - iterable: strings, lists, tuples, dictionaries, and sets are all iterable objects (containers) which we can get an iterator from. A range() functiuon also returns iterable object. All these objects have an iter() method which is used to get an iterator.


If we comment out the "yield" line, we get "TypeError: NoneType object is not an iterator" at the next(doc_stream) in "get_minibatch()" function.



Output from the code:

X_train, y_train= ['line-a', 'line-b', 'line-c'] [' 1', ' 2', ' 3']
X_train, y_train= ['line-d', 'line-e', 'line-f'] [' 4', ' 5', ' 6']
X_train, y_train= ['line-g', 'line-h', 'line-i'] [' 7', ' 8', ' 9']
X_train, y_train= ['line-j ', 'line-k ', 'line-k '] ['10', '11', '12']

The input used in the code looks like this:

line-a 1
line-b 2
line-c 3
line-d 4
line-e 5
line-f 6
line-g 7
line-h 8
line-i 9
line-j 10
line-k 11
line-k 12

The digit at the end of each line is used as a class label(y_train), so we want to keep it separate from the rest of the text(X_train).




For more information about yield or generator, please visit:

  1. The yield keyword
  2. Generator Functions and Expressions





What is __init__.py?

It is used to import a module in a directory, which is called package import.

If we have a module, dir1/dir2/mod.py, we put __init__.py in each directories so that we can import the mod like this:

import dir1.dir2.mod

The __init__.py is usually an empty py file. The hierarchy gives us a convenient way of organizing the files in a large system.





Build a string with the numbers from 0 to 100, "0123456789101112..."

We may want to use str.join rather than appending a number every time.

>>> ''.join([`x` for x in xrange(101)])
'0123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100'
>>> 

Note that the (`) is a backtick not a regiular single quote ('):

>>> type(1)
<type 'int'>
>>> type(`1`)
<type 'str'>


Also tote that we cannot use double quote(") or single quote(') to make n a string:

>>> type("1")
<type 'str'>

>>> ''.join(["n" for n in range(10)])
'nnnnnnnnnn'
>>> ''.join(['n' for n in range(10)])
'nnnnnnnnnn'

>>> n = 1
>>> print `n`
1
>>> print "n"
n

Warning: the usage of the backtick(`) in the example above has been deprecated and no longer works in Python3. So, we may want to use str() instead as shown in the example below.


Note: join() returns a string in which the string elements of sequence have been joined by string separator.

We can use str(x) instead:

>>> ''.join([str(x) for x in range(10)])
'0123456789'

Also, since the xrange() is replaced with range in Python 3.x, we should use range() instead for compatibility.




Basic file processing: Printing contents of a file.
try:
    with open('filename','r') as f:
        print f.read()
except IOError:
    print "No such file exists"




How can we get home directory using '~' in Python?

We need to import os module, and add just one line:

import os
print os.path.expanduser('~')

Output:

/home/k




The usage of os.path.dirname() & os.path.basename()

For example, we have the path like this, /home/k/TEST/PYTHON/p.py:

We can get the dir and file using the following:

First, we need to import os module:

>>> import os

Then, we do:

>>> os.path.dirname('/home/k/TEST/PYTHON/p.py')
'/home/k/TEST/PYTHON'

>>> os.path.basename('/home/k/TEST/PYTHON/p.py')
'p.py'

Or we can get them at once in tuple using os.path.split():

>>> os.path.split('/home/k/TEST/PYTHON/p.py')
('/home/k/TEST/PYTHON', 'p.py')

If we want to combine and make a full path:

>>> os.path.join('/home/k/TEST/PYTHON', 'p.py')
'/home/k/TEST/PYTHON/p.py'






Default Libraries

We should be able to answer the questions about the standard library.
Such as "Do you know if there's a standard library for recursive file renaming?",
or "In which library would you use for regular expression?"

  1. os: operating system support
    os.path: Common pathname manipulations:
    >>> import os
    >>> print(os.getcwd())
    C:\Python32
    >>> cur_dir = os.curdir
    >>> print(cur_dir)
    .
    >>> scripts_dir = os.path.join(os.curdir, 'Tools\Scripts')
    >>> print(scripts_dir)
    .\Tools\Scripts
    >>> diff_py = os.path.join(scripts_dir, 'diff.py')
    >>> print(diff_py)
    .\Tools\Scripts\diff.py
    >>> os.path.basename(diff_py)
    ' diff.py'
    >>> os.path.splitext(diff_py)
    ('.\\Tools\\Scripts\\diff', '.py')
    

    The os.path.join() function constructs a pathname out of one or more partial pathnames. In this case, it simply concatenates strings.

    Other convenient ones are: dirname() and basename(), which are the 1st and 2nd element of split(), respectively:

    >>> import os
    >>> print(os.getcwd())
    C:\TEST\dirA\dirB\dirC
    >>> print(os.path.dirname(os.getcwd()))
    C:\TEST\dirA\dirB
    >>> print(os.path.basename(os.getcwd()))
    dirC
    >>> print(os.path.split(os.getcwd()))
    ('C:\\TEST\\dirA\\dirB', 'dirC')
    >>> pathname = os.path.join(os.getcwd(),'myfile.py')
    >>> pathname
    'C:\\TEST\\dirA\\dirB\\dirC\\myfile.py'
    >>> (dirname, filename) = os.path.split(pathname)
    >>> dirname
    'C:\\TEST\\dirA\\dirB\\dirC'
    >>> filename
    'myfile.py'
    

    The split function splits a full pathname and returns a tuple containing the path and filename. We could use multi-variable assignment to return multiple values from a function. The os.path.split() function does exactly that. We assign the return value of the split function into a tuple of two variables. Each variable receives the value of the corresponding element of the returned tuple.

    The first variable, dirname, receives the value of the first element of the tuple returned from the os.path.split() function, the file path. The second variable, filename, receives the value of the second element of the tuple returned from the os.path.split() function, the filename.

    os.path also contains the os.path.splitext() function, which splits a filename and returns a tuple containing the filename and the file extension.

    The os.path.expanduser() function :

    >>> print(os.path.expanduser('~'))
    C:\Users\KHong
    

    will expand a pathname that uses ~ to represent the current user's home directory. This works on any platform where users have a home directory, including Linux, Mac OS X, and Windows. The returned path does not have a trailing slash, but the os.path.join() function doesn't mind:

  2. re: Regular expression operations
    Visit Regular Expressions with Python

  3. itertools: Functions creating iterators for efficient looping.
    It includes permutations, combinations and other useful iterables.
    >>> [x for x in itertools.permutations('123')]
    [('1', '2', '3'), ('1', '3', '2'), ('2', '1', '3'), ('2', '3', '1'), ('3', '1', '2'), ('3', '2', '1')]
    >>> [x for x in itertools.permutations('123',2)]
    [('1', '2'), ('1', '3'), ('2', '1'), ('2', '3'), ('3', '1'), ('3', '2')]
    >>> 
    






range vs xrange
>>> sum(range(1,101))
5050
>>> sum(xrange(1,101))
5050
>>> 

range() returns a list to the sum function containing all the numbers from 1 to 100. But xrange() returns an iterator rather than a list, which makes it more lighter in terms of memory use as shown below.

>>> range(1,5)
[1, 2, 3, 4]
>>> xrange(1,5)
xrange(1, 5)
>>> 

Note: Since the xrange() is replaced with range in Python 3.x, we should use range() instead for compatibility. The range() in Python 3.x just returns iterator. That means it does not produce the results in memory any more, and if we want to get list from range(), we need to force it to do so: list(range(...)).





Iterators

Python defines several iterator objects to support iteration over general and specific sequence types, dictionaries.

Iterable:
  1. An iterable is any Python object capable of returning its elements one at a time. Examples of iterables include sequences like lists, tuples, and strings, as well as other iterable objects like sets and dictionaries.
  2. Iterable objects can be used in a for loop because they can produce an iterator.
  3. We can check if an object is iterable using the iter() function and catching a TypeError if it's not iterable.
  4. my_list = [1, 2, 3]
    iter_obj = iter(my_list)
    
    for element in iter_obj:
        print(element)    
    

Iterator:
  1. An iterator is an object representing a stream of data. It must implement two methods: __iter__() and __next__().
  2. The __iter__() method returns the iterator object itself, and the __next__() method returns the next element from the stream.
  3. When there are no more elements, the __next__() method should raise the StopIteration exception.
  4. We can create an iterator from an iterable using the iter() function.
  5. my_list = [1, 2, 3]
    iter_obj = iter(my_list)
    
    print(next(iter_obj))  # Output: 1
    print(next(iter_obj))  # Output: 2
    

Any object with a __next__() method to advance to a next result is considered iterator. Note that if an object has __iter__() method, we call the object iterable.

iterators-iterables.png
a = [1,2,3]

try:
    iterator = iter(a)
    print(iterator)
    print(iterator.__next__())  # for python 2, use next() instead
    print(iterator.__next__())
    print(iterator.__next__())
    print(iterator.__next__())
except StopIteration:
    print("StopIteration")    

Output:

<list_iterator object at 0x108ac0610>
1
2
3
StopIteration    

For more info, please visit
http://www.bogotobogo.com/python/python_iterators.php.






Generators

Generators allow us to declare a function that behaves like an iterator, i.e. it can be used in a for loop. It's a function type generator, but there is another type of generator that may be more familiar to us - expression type generator used in list comprehension:

>>> # List comprehension makes a list
>>> [ x ** 3 for x in range(5)]
[0, 1, 8, 27, 64]
>>> 
>>> # Generator expression makes an iterable
>>> (x ** 3 for x in range(5))
<generator object <genexpr> at 0x000000000315F678>

With the generator expression, we can just wrap it with list() call:

>>> list(x ** 3 for x in range(5))
[0, 1, 8, 27, 64]

Since every generator is an iterator, we can use next() to get the values:

>>> generator = (x ** 3 for x in range(5))
>>> generator.__next__()   # for python2, use next() instead
0
>>> generator.__next__()
1
>>> generator.__next__()
8
>>> generator.__next__()
27



For more information about generators, please visit:
http://www.bogotobogo.com/python/python_generators.php





Manipulating functions as first-class objects

Functions as first-class objects?
That means we can pass them around as objects and can manipulate them. In other words, most of the times, this just means we can pass these first-class citizens as arguments to functions, or return them from functions. Everything in Python is a proper object. Even things that are "primitive types" in other languages:

>>> dir (100)
['__abs__', '__add__', '__and__', '__class__', '__cmp__', '__coerce__', 
'__delattr__', '__div__', '__divmod__', '__doc__', '__float__', 
....
 'numerator', 'real']
>>> 

Functions have attributes and can be referenced and assigned to variables.

>>> def one(arg):
	'''I am a function returning arg I received.'''
	return arg

>>> one(1)
1
>>> one
<function one at 0x0284AA70>
>>> One = one
>>> One
<function one at 0x0284AA70>
>>> one.__doc__
'I am a function returning arg I received.'
>>> 






docstrings vs comments

A docstring is the documentation string for a function. We use it as shown below:

function_name.__doc__

We can declare it like this:

def my_function():
    """our docstring"""

or:

def my_function():
    '''our docstring'''

Everything between the triple quotes (with double quotes, """ or with single quotes,''') is the function's docstring, which documents what the function does. A docstring, if it exists, must be the first thing defined in a function. In other words, it should appear on the next line after the function declaration. We don't technically need to give our function a docstring, but we always should. The docstring will be available at runtime as an attribute of the function.

Writing documentation for our program this way makes the code more readable. We can also use comments for clarification of what the code is doing. In general, docstrings are for documentation, comments are for a code reader.





Monkey-patching

The origin of monkey-patch according to wiki is :
"The term monkey patch seems to have come from an earlier term, guerrilla patch, which referred to changing code sneakily at runtime. The word guerrilla, homophonous with gorilla, became monkey, possibly to make the patch sound less intimidating."

In Python, the term monkey patch only refers to dynamic modifications of a class or module at runtime, motivated by the intent to patch existing third-party code as a workaround to a bug or feature which does not act as we desire.

We have a module called m.py like this:

# m.py
class MyClass:
    def f(self):
        print "f()"

Then, if we run the monkey-patch testing like this:

>>> import m
>>> def monkey_f(self):
	print "monkey_f()"

	
>>> m.MyClass.f = monkey_f
>>> obj = m.MyClass()
>>> obj.f()
monkey_f()
>>> 

As we can see, we did make some changes in the behavior of f() in MyClass using the function we defined, monkey_f(), outside of the module m.

It is a risky thing to do, but sometimes we need this trick, such as testing.





pdb - The Python Debugger

The module pdb defines an interactive source code debugger for Python programs. It supports setting (conditional) breakpoints and single stepping at the source line level, inspection of stack frames, source code listing, and evaluation of arbitrary Python code in the context of any stack frame. It also supports post-mortem debugging and can be called under program control.




Using Lambda

Python supports the creation of anonymous functions (i.e. functions that are not bound to a name) at runtime, using a construct called lambda. This is not exactly the same as lambda in functional programming languages such as Lisp, but it is a very powerful concept that's well integrated into Python and is often used in conjunction with typical functional concepts like filter(), map() and reduce().

The following code shows the difference between a normal function definition, func and a lambda function, lamb:

>>> 
>>> def func(x): return x ** 3

>>> print(func(5))
125
>>> 
>>> lamb = lambda x: x ** 3
>>> print(lamb(5))
125
>>> 

As we can see, func() and lamb() do exactly the same and can be used in the same ways. Note that the lambda definition does not include a return statement -- it always contains an expression which is returned. Also note that we can put a lambda definition anywhere a function is expected, and we don't have to assign it to a variable at all.

The lambda's general form is :

lambda arg1, arg2, ...argN : expression using arguments

Function objects returned by running lambda expressions work exactly the same as those created and assigned by defs. However, there are a few differences that make lambda useful in specialized roles:

  1. lambda is an expression, not a statement.
    Because of this, a lambda can appear in places a def is not allowed. For example, places like inside a list literal, or a function call's arguments. As an expression, lambda returns a value that can optionally be assigned a name. In contrast, the def statement always assigns the new function to the name in the header, instead of returning is as a result.
  2. lambda's body is a single expression, not a block of statements.
    The lambda's body is similar to what we'd put in a def body's return statement. We simply type the result as an expression instead of explicitly returning it. Because it is limited to an expression, a lambda is less general that a def. We can only squeeze design, to limit program nesting. lambda is designed for coding simple functions, and def handles larger tasks.
>>> 
>>> def f(x, y, z): return x + y + z

>>> f(2, 30, 400)
432

We can achieve the same effect with lambda expression by explicitly assigning its result to a name through which we can call the function later:

>>> 
>>> f = lambda x, y, z: x + y + z
>>> f(2, 30, 400)
432
>>> 

Here, the function object the lambda expression creates is assigned to f. This is how def works, too. But in def, its assignment is an automatic must.

For more, please go to Functions lambda







Properties vs Getters/Setters

We have more detailed discussion in Classes and Instances - Method: Properties.

In general, properties are more flexible than attributes. That's because we can define functions that describe what is supposed to happen when we need setting, getting or deleting them. If we don't need this additional flexibility, we may just use attributes since they are easier to declare and faster.

However, when we convert an attribute into a property, we just define some getter and setter that we attach to it, that will hook the data access. Then, we don't need to rewrite the rest of our code, the way for accessing the data is the same, whatever our attribute is a property or not.






classmethod vs staticmethod

From the official Python documentation on @classmethod:

classmethod(function)
    Return a class method for function.

    A class method receives the class as implicit first argument, 
    just like an instance method receives the instance. 
    To declare a class method, use this idiom:

class C:
    @classmethod
    def f(cls, arg1, arg2, ...): ...

The @classmethod form is a function decorator.
It can be called either on the class (such as C.f()) or on an instance (such as C().f()). 
The instance is ignored except for its class. If a class method is called for a derived class, 
the derived class object is passed as the implied first argument.

And for the @staticmethod, the python doc describes it as below:

staticmethod(function)
    Return a static method for function.

    A static method does not receive an implicit first argument. 
    To declare a static method, use this idiom:

class C:
    @staticmethod
    def f(arg1, arg2, ...): ...

The @staticmethod form is a function decorator.
It can be called either on the class (such as C.f()) or on an instance (such as C().f()). 
The instance is ignored except for its class.
Static methods in Python are similar to those found in Java or C++.

For more info on static vs class methods, please visit:

@static method vs class method





Making a list with unique element from a list with duplicate elements

Iterating the list is not a desirable solution. The right answer should look like this:

>>> dup_list = [1,2,3,4,4,4,5,1,2,7,8,8,10]
>>> unique_list = list(set(dup_list))
>>> print unique_list
[1, 2, 3, 4, 5, 7, 8, 10]
>>> 

Or without using the set:

dup_list = [1,2,3,4,4,4,5,1,2,7,8,8,10]

#d = {}.fromkeys(dup_list,0) 
d = {k:0 for k in dup_list}
print(d)

for x in dup_list:
    d[x] += 1
print(d)

uniq = [ k for k,v in d.items() if v == 1]
print(uniq)    

Output:

{1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 7: 0, 8: 0, 10: 0}
{1: 2, 2: 2, 3: 1, 4: 3, 5: 1, 7: 1, 8: 2, 10: 1}
[3, 5, 7, 10]    



Name the functional approach that Python is taking.

Python provides the following:

  1. map(aFunction, aSequence)
  2. filter(aFunction, aSequence)
  3. reduce(aFunction, aSequence)
  4. lambda
  5. list comprehension

These functions are all convenience features in that they can be written in Python fairly easily. Functional programming is all about expressions. We may say that the Functional programming is an expression oriented programming.




What is map?

The syntax of map is:

map(aFunction, aSequence)

The first argument is a function to be executed for all the elements of the iterable given as the second argument. If the function given takes in more than 1 arguments, then many iterables are given.

>>> def cubic(x):
	return x*x*x

>>> items = [x for x in range(11) if x % 2 == 0]
>>> list(map(cubic, items))
[0, 8, 64, 216, 512, 1000]
>>>
>>> list(map(lambda x,y: x*y, [1,2,3], [4,5,6]))
[4, 10, 18]
>>> 

map is similar to list comprehension but is more limited because it requires a function instead of an arbitrary expression.





What is filter and reduce?

Just for comparison purpose, in the following example, we will include map as well.

>>> integers = [ x for x in range(11)]
>>> filter(lambda x: x % 2 == 0, integers)
[0, 2, 4, 6, 8, 10]
>>> map(lambda x: x**2, integers)
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
>>> reduce(lambda x, y: x + y, integers)
55
>>> 

In the above example, we defined a simple list of integer values, then we use the standard functions filter(), map() and reduce() to do various things with that list. All of the three functions expect two arguments: A function and a list.
In the first example, filter() calls our lambda function for each element of the list, and returns a new list that contains only those elements for which the function returned "True". In this case, we get a list of all even numbers.
In the second example, map() is used to convert our list. The given function is called for every element in the original list, and a new list is created which contains the return values from our lambda function. In this case, it computes x^2 for every element.
Finally, reduce() is somewhat special. The function for this one must accept two arguments (x and y), not just one. The function is called with the first two elements from the list, then with the result of that call and the third element, and so on, until all of the list elements have been handled. This means that our function is called n-1 times if the list contains n elements. The return value of the last call is the result of the reduce() construct. In the above example, it simply adds the arguments, so we get the sum of all elements.





*args and **kwargs

Putting *args and/or **kwargs as the last items in our function definition's argument list allows that function to accept an arbitrary number of anonymous and/or keyword arguments.
Those arguments are called Keyword Arguments. Actually, they are place holders for multiple arguments, and they are useful especially when we need to pass a different number of arguments each time we call the function.

We may want to use *args when we're not sure how many arguments might be passed to our function, i.e. it allows us to pass an arbitrary number of arguments to your function as shown in the example below:

Let's make a function that sums of all numbers. It should work for bo the inputs: 1,2,3,4,5 as separate args and as a list [1,2,3,4,5]:

def fn(*args):
  ans = 0 
  for x in args:
    ans += x
return ans

print(fn(1,2,3,4,5))
print(fn(*[1, 2, 3, 4, 5]))
print(fn(*range(1,6)))

Note here if we give the list itself instead unpacking('*'), we get an error:

print(fn([1, 2, 3, 4, 5]))    

The error we get is:

TypeError: unsupported operand type(s) for +=: 'int' and 'list'    

That's because we're trying to do 0 + [1,2,3,4,5] which is not allowed.

So, we need to check how the arg is passed to the function with the following code:

def fn(*args):
  print('args={0}'.format(args))

fn(1,2,3,4,5)       # args=(1, 2, 3, 4, 5)
fn(*[1,2,3,4,5])    # args=(1, 2, 3, 4, 5)
fn([1,2,3,4,5])     # args=([1, 2, 3, 4, 5],)

As we can see, fn(*[1,2,3,4,5]) is passing args uppacked, while fn([1,2,3,4,5]) is passing a list as one parameter.



The keyword arguments is a special name=value syntax in function calls that specifies passing by name. It is often used to provide configuration options.

>>> def kargs_function(**kargs):
	  for k,v in kargs.items():
		print (k,v)

>>> kargs_function(**{'uno':'one','dos':'two','tres':'three'})
('dos', 'two')
('tres', 'three')
('uno', 'one')
>>>
>>> kargs_function(dos='two', tres='three', uno='one')
('dos', 'two')
('tres', 'three')
('uno', 'one')
>>> 

For more details, please visit *args and **kwargs - Collecting and Unpacking Arguments.







mutable vs immutable

The content of objects of immutable types cannot be changed after they are created.

immutable mutable
tuple, frozen set, int, float, str list, set, dict, byte array





Difference between remove, del and pop on lists

To remove a list element, we can use either the del statement if we know exactly which element(s) we are deleting or the remove() method if we do not know.

list.remove(element), del list(index), list.pop(index)

remove() removes the first matching value, not a specific index:

>>> a = [5,6,7,7,8]
>>> a.remove(7)
>>> a
[5, 6, 7, 8]

Both del and pop() work on index while the pop() returns an element:

>>> a = [5,6,7,7,8]
>>> del a[1]
>>> a
[5, 7, 7, 8]

>>> a = [5,6,7,7,8]
>>> a.pop(1)
6
>>> a
[5, 7, 7, 8]

>>> a = [5,6,7,7,8]
>>> a.pop(a.index(6)) # get the index for 6
6
>>> a
[5, 7, 7, 8]




Join with new line

Given a list of string, ['Black', 'holes', 'are', 'where', 'God', 'divided', 'by', 'zero'], print each word in a new line:

>>> s = ['Black', 'holes', 'are', 'where', 'God', 'divided', 'by', 'zero']
>>> print '\n'.join(s)
Black
holes
are
where
God
divided
by
zero











  1. Python Coding Questions I
  2. Python Coding Questions II
  3. Python Coding Questions III
  4. Python Coding Questions IV
  5. Python Coding Questions V
  6. Python Coding Questions VI
  7. Python Coding Questions VII
  8. Python Coding Questions VIII
  9. Python Coding Questions IX
  10. Python Coding Questions X





List of codes Q & A

  1. Merging two sorted list
  2. Get word frequency - initializing dictionary
  3. Initializing dictionary with list
  4. map, filter, and reduce
  5. Write a function f() - yield
  6. What is __init__.py?
  7. Build a string with the numbers from 0 to 100, "0123456789101112..."
  8. Basic file processing: Printing contents of a file - "with open"
  9. How can we get home directory using '~' in Python?
  10. The usage of os.path.dirname() & os.path.basename() - os.path
  11. Default Libraries
  12. range vs xrange
  13. Iterators
  14. Generators
  15. Manipulating functions as first-class objects
  16. docstrings vs comments
  17. using lambdda
  18. classmethod vs staticmethod
  19. Making a list with unique element from a list with duplicate elements
  20. What is map?
  21. What is filter and reduce?
  22. *args and **kwargs
  23. mutable vs immutable
  24. Difference between remove, del and pop on lists
  25. Join with new line
  26. Hamming distance
  27. Floor operation on integers
  28. Fetching every other item in the list
  29. Python type() - function
  30. Dictionary Comprehension
  31. Sum
  32. Truncating division
  33. Python 2 vs Python 3
  34. len(set)
  35. Print a list of file in a directory
  36. Count occurrence of a character in a Python string
  37. Make a prime number list from (1,100)
  38. Reversing a string - Recursive
  39. Reversing a string - Iterative
  40. Reverse a number
  41. Output?
  42. Merging overlapped range
  43. Conditional expressions (ternary operator)
  44. Packing Unpacking
  45. Function args
  46. Unpacking args
  47. Finding the 1st revision with a bug
  48. Which one has higher precedence in Python? - NOT, AND , OR
  49. Decorator(@) - with dollar sign($)
  50. Multi-line coding
  51. Recursive binary search
  52. Iterative binary search
  53. Pass by reference
  54. Simple calculator
  55. iterator class that returns network interfaces
  56. Converting domain to ip
  57. How to count the number of instances
  58. Python profilers - cProfile
  59. Calling a base class method from a child class that overrides it
  60. How do we find the current module name?
  61. Why did changing list 'newL' also change list 'L'?
  62. Constructing dictionary - {key:[]}
  63. Colon separated sequence
  64. Converting binary to integer
  65. 9+99+999+9999+...
  66. Calculating balance
  67. Regular expression - findall
  68. Chickens and pigs
  69. Highest possible product
  70. Implement a queue with a limited size
  71. Copy an object
  72. Filter
  73. Products
  74. Pickle
  75. Overlapped Rectangles
  76. __dict__
  77. Fibonacci I - iterative, recursive, and via generator
  78. Fibonacci II - which method?
  79. Fibonacci III - find last two digits of Nth Fibonacci number
  80. Write a Stack class returning Max item at const time A
  81. Write a Stack class returning Max item at const time B
  82. Finding duplicate integers from a list - 1
  83. Finding duplicate integers from a list - 2
  84. Finding duplicate integers from a list - 3
  85. Reversing words 1
  86. Parenthesis, a lot of them
  87. Palindrome / Permutations
  88. Constructing new string after removing white spaces
  89. Removing duplicate list items
  90. Dictionary exercise
  91. printing numbers in Z-shape
  92. Factorial
  93. lambda
  94. lambda with map/filter/reduce
  95. Number of integer pairs whose difference is K
  96. iterator vs generator
  97. Recursive printing files in a given directory
  98. Bubble sort
  99. What is GIL (Global Interpreter Lock)?
  100. Word count using collections
  101. Pig Latin
  102. List of anagrams from a list of words
  103. lamda with map, filer and reduce functions
  104. Write a code sending an email using gmail
  105. histogram 1 : the frequency of characters
  106. histogram 2 : the frequency of ip-address
  107. Creating a dictionary using tuples
  108. Getting the index from a list
  109. Looping through two lists side by side
  110. Dictionary sort with two keys : primary / secondary keys
  111. Writing a file downloaded from the web
  112. Sorting csv data
  113. Reading json file
  114. Sorting class objects
  115. Parsing Brackets
  116. Printing full path
  117. str() vs repr()
  118. Missing integer from a sequence
  119. Polymorphism
  120. Product of every integer except the integer at that index
  121. What are accessors, mutators, and @property?
  122. N-th to last element in a linked list
  123. Implementing linked list
  124. Removing duplicate element from a list
  125. List comprehension
  126. .py vs .pyc
  127. Binary Tree
  128. Print 'c' N-times without a loop
  129. Quicksort
  130. Dictionary of list
  131. Creating r x c matrix
  132. Transpose of a matrix
  133. str.isalpha() & str.isdigit()
  134. Regular expression
  135. What is Hashable? Immutable?
  136. Convert a list to a string
  137. Convert a list to a dictionary
  138. List - append vs extend vs concatenate
  139. Use sorted(list) to keep the original list
  140. list.count()
  141. zip(list,list) - join elements of two lists
  142. zip(list,list) - weighted average with two lists
  143. Intersection of two lists
  144. Dictionary sort by value
  145. Counting the number of characters of a file as One-Liner
  146. Find Armstrong numbers from 100-999
  147. Find GCF (Greatest common divisor)
  148. Find LCM (Least common multiple)
  149. Draws 5 cards from a shuffled deck
  150. Dictionary order by value or by key
  151. Regular expression - re.split()
  152. Regular expression : re.match() vs. re.search()
  153. Regular expression : re.match() - password check
  154. Regular expression : re.search() - group capturing
  155. Regular expression : re.findall() - group capturin
  156. Prime factors : n = products of prime numbers
  157. Valid IPv4 address
  158. Sum of strings
  159. List rotation - left/right
  160. shallow/deep copy
  161. Converting integer to binary number
  162. Creating a directory and a file
  163. Creating a file if not exists
  164. Invoking a python file from another
  165. Sorting IP addresses
  166. Word Frequency
  167. Printing spiral pattern from a 2D array - I. Clock-wise
  168. Printing spiral pattern from a 2D array - II. Counter-Clock-wise
  169. Find a minimum integer not in the input list
  170. I. Find longest sequence of zeros in binary representation of an integer
  171. II. Find longest sequence of zeros in binary representation of an integer - should be surrounded with 1
  172. Find a missing element from a list of integers
  173. Find an unpaired element from a list of integers
  174. Prefix sum : Passing cars
  175. Prefix sum : count the number of integers divisible by k in range [A,B]
  176. Can make a triangle?
  177. Dominant element of a list
  178. Minimum perimeter
  179. MinAbsSumOfTwo
  180. Ceiling - Jump Frog
  181. Brackets - Nested parentheses
  182. Brackets - Nested parentheses of multiple types
  183. Left rotation - list shift
  184. MaxProfit
  185. Stack - Fish
  186. Stack - Stonewall
  187. Factors or Divisors
  188. String replace in files 1
  189. String replace in files 2
  190. Using list as the default_factory for defaultdict
  191. Leap year
  192. Capitalize
  193. Log Parsing
  194. Getting status_code for a site
  195. 2D-Array - Max hourglass sum
  196. New Year Chaos - list
  197. List (array) manipulation - list
  198. Hash Tables: Ransom Note
  199. Count Triplets with geometric progression
  200. Strings: Check if two strings are anagrams
  201. Strings: Making Anagrams
  202. Strings: Alternating Characters
  203. Special (substring) Palindrome
  204. String with the same frequency of characters
  205. Common Child
  206. Fraudulent Activity Notifications
  207. Maximum number of toys
  208. Min Max Riddle
  209. Poisonous Plants with Pesticides
  210. Common elements of 2 lists - Complexity
  211. Get execution time using decorator(@)
  212. Conver a string to lower case and split using decorator(@)
  213. Python assignment and memory location
  214. shallow copy vs deep copy for compound objects (such as a list)
  215. Generator with Fibonacci
  216. Iterator with list
  217. Second smallest element of a list
  218. *args, **kargs, and positional args
  219. Write a function, fn('x','y',3) that returns ['x1', 'y1', 'x2', 'y2', 'x3', 'y3']
  220. sublist or not
  221. any(), all()
  222. Flattening a list
  223. Select an element from a list
  224. Circularly identical lists
  225. Difference between two lists
  226. Reverse a list
  227. Split a list with a step
  228. Break a list and make chunks of size n
  229. Remove duplicate consecutive elements from a list
  230. Combination of elements from two lists
  231. Adding a sublist
  232. Replace the first occurence of a value
  233. Sort the values of the first list using the second list
  234. Transpose of a matrix (nested list)
  235. Binary Gap
  236. Powerset
  237. Round Robin
  238. Fixed-length chunks or blocks
  239. Accumulate
  240. Dropwhile
  241. Groupby
  242. Simple product
  243. Simple permutation
  244. starmap(fn, iterable)
  245. zip_longest(*iterables, fillvalue=None)
  246. What is the correct way to write a doctest?
  247. enumerate(iterable, start=0)
  248. collections.defaultdict - grouping a sequence of key-value pairs into a dictionary of lists
  249. What is the purpose of the 'self' keyword when defining or calling instance methods?
  250. collections.namedtuple(typename, field_names, *, rename=False, defaults=None, module=None)
  251. zipped
  252. What is key difference between a set and a list?
  253. What does a class's init() method do?
  254. Class methods
  255. Permutations and combinations of ['A','B','C']
  256. Sort list of dictionaries by values
  257. Return a list of unique words
  258. hashlib
  259. encode('utf-8')
  260. Reading in CSV file
  261. Count capital letters in a file
  262. is vs ==
  263. Create a matrix : [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
  264. Binary to integer and check if it's the power of 2
  265. urllib.request.urlopen() and requests
  266. Game statistics
  267. Chess - pawn race
  268. Decoding a string
  269. Determinant of a matrix - using numpy.linalg.det()
  270. Revenue from shoe sales - using collections.Counter()
  271. Rangoli
  272. Unique characters
  273. Valid UID
  274. Permutations of a string in lexicographic sorted order
  275. Nested list
  276. Consecutive digit count
  277. Find a number that occurs only once
  278. Sorting a two-dimensional array
  279. Reverse a string
  280. Generate random odd numbers in a range
  281. Shallow vs Deep copy
  282. Transpose matrix
  283. Are Arguments in Python Passed by Value or by Reference?
  284. re: Is a string alphanumeric?
  285. reversed()
  286. Caesar's cipher, or shift cipher, Caesar's code, or Caesar shift
  287. Every other words
  288. re: How can we check if an email address is valid or not?
  289. re: How to capture temperatures of a text
  290. re.split(): How to split a text.
  291. How can we merge two dictionaries?
  292. How can we combine two dictionaries?
  293. What is the difference between a generator and a list?
  294. Pairs of a given array A whose sum value is equal to a target value N
  295. Adding two integers without plus
  296. isinstance() vs type()
  297. What is a decorator?
  298. In Python slicing, what does my_list[-3:2:-2] slice do?
  299. Revisit sorting dict - counting chars in a text file
  300. re: Transforming a date format using re.sub
  301. How to replace the newlines in csv file with tabs?
  302. pandas.merge
  303. How to remove duplicate charaters from a string?
  304. Implement a class called ComplexNumber
  305. Find a word frequency
  306. Get the top 3 most frequent characters of a string
  307. Just seen and ever seen
  308. Capitalizing the full name
  309. Counting Consequitive Characters
  310. Calculate Product of a List of Integers Provided using input()
  311. How many times a substring appears in a string
  312. Hello, first_name last_name
  313. String validators
  314. Finding indices that a char occurs in a list
  315. itertools combinations









Python tutorial



Python Home

Introduction

Running Python Programs (os, sys, import)

Modules and IDLE (Import, Reload, exec)

Object Types - Numbers, Strings, and None

Strings - Escape Sequence, Raw String, and Slicing

Strings - Methods

Formatting Strings - expressions and method calls

Files and os.path

Traversing directories recursively

Subprocess Module

Regular Expressions with Python

Regular Expressions Cheat Sheet

Object Types - Lists

Object Types - Dictionaries and Tuples

Functions def, *args, **kargs

Functions lambda

Built-in Functions

map, filter, and reduce

Decorators

List Comprehension

Sets (union/intersection) and itertools - Jaccard coefficient and shingling to check plagiarism

Hashing (Hash tables and hashlib)

Dictionary Comprehension with zip

The yield keyword

Generator Functions and Expressions

generator.send() method

Iterators

Classes and Instances (__init__, __call__, etc.)

if__name__ == '__main__'

argparse

Exceptions

@static method vs class method

Private attributes and private methods

bits, bytes, bitstring, and constBitStream

json.dump(s) and json.load(s)

Python Object Serialization - pickle and json

Python Object Serialization - yaml and json

Priority queue and heap queue data structure

Graph data structure

Dijkstra's shortest path algorithm

Prim's spanning tree algorithm

Closure

Functional programming in Python

Remote running a local file using ssh

SQLite 3 - A. Connecting to DB, create/drop table, and insert data into a table

SQLite 3 - B. Selecting, updating and deleting data

MongoDB with PyMongo I - Installing MongoDB ...

Python HTTP Web Services - urllib, httplib2

Web scraping with Selenium for checking domain availability

REST API : Http Requests for Humans with Flask

Blog app with Tornado

Multithreading ...

Python Network Programming I - Basic Server / Client : A Basics

Python Network Programming I - Basic Server / Client : B File Transfer

Python Network Programming II - Chat Server / Client

Python Network Programming III - Echo Server using socketserver network framework

Python Network Programming IV - Asynchronous Request Handling : ThreadingMixIn and ForkingMixIn

Python Coding Questions I

Python Coding Questions II

Python Coding Questions III

Python Coding Questions IV

Python Coding Questions V

Python Coding Questions VI

Python Coding Questions VII

Python Coding Questions VIII

Python Coding Questions IX

Python Coding Questions X

Image processing with Python image library Pillow

Python and C++ with SIP

PyDev with Eclipse

Matplotlib

Redis with Python

NumPy array basics A

NumPy Matrix and Linear Algebra

Pandas with NumPy and Matplotlib

Celluar Automata

Batch gradient descent algorithm

Longest Common Substring Algorithm

Python Unit Test - TDD using unittest.TestCase class

Simple tool - Google page ranking by keywords

Google App Hello World

Google App webapp2 and WSGI

Uploading Google App Hello World

Python 2 vs Python 3

virtualenv and virtualenvwrapper

Uploading a big file to AWS S3 using boto module

Scheduled stopping and starting an AWS instance

Cloudera CDH5 - Scheduled stopping and starting services

Removing Cloud Files - Rackspace API with curl and subprocess

Checking if a process is running/hanging and stop/run a scheduled task on Windows

Apache Spark 1.3 with PySpark (Spark Python API) Shell

Apache Spark 1.2 Streaming

bottle 0.12.7 - Fast and simple WSGI-micro framework for small web-applications ...

Flask app with Apache WSGI on Ubuntu14/CentOS7 ...

Fabric - streamlining the use of SSH for application deployment

Ansible Quick Preview - Setting up web servers with Nginx, configure enviroments, and deploy an App

Neural Networks with backpropagation for XOR using one hidden layer

NLP - NLTK (Natural Language Toolkit) ...

RabbitMQ(Message broker server) and Celery(Task queue) ...

OpenCV3 and Matplotlib ...

Simple tool - Concatenating slides using FFmpeg ...

iPython - Signal Processing with NumPy

iPython and Jupyter - Install Jupyter, iPython Notebook, drawing with Matplotlib, and publishing it to Github

iPython and Jupyter Notebook with Embedded D3.js

Downloading YouTube videos using youtube-dl embedded with Python

Machine Learning : scikit-learn ...

Django 1.6/1.8 Web Framework ...




Ph.D. / Golden Gate Ave, San Francisco / Seoul National Univ / Carnegie Mellon / UC Berkeley / DevOps / Deep Learning / Visualization

YouTubeMy YouTube channel

Sponsor Open Source development activities and free contents for everyone.

Thank you.

- K Hong







Python tutorial



Python Home

Introduction

Running Python Programs (os, sys, import)

Modules and IDLE (Import, Reload, exec)

Object Types - Numbers, Strings, and None

Strings - Escape Sequence, Raw String, and Slicing

Strings - Methods

Formatting Strings - expressions and method calls

Files and os.path

Traversing directories recursively

Subprocess Module

Regular Expressions with Python

Regular Expressions Cheat Sheet

Object Types - Lists

Object Types - Dictionaries and Tuples

Functions def, *args, **kargs

Functions lambda

Built-in Functions

map, filter, and reduce

Decorators

List Comprehension

Sets (union/intersection) and itertools - Jaccard coefficient and shingling to check plagiarism

Hashing (Hash tables and hashlib)

Dictionary Comprehension with zip

The yield keyword

Generator Functions and Expressions

generator.send() method

Iterators

Classes and Instances (__init__, __call__, etc.)

if__name__ == '__main__'

argparse

Exceptions

@static method vs class method

Private attributes and private methods

bits, bytes, bitstring, and constBitStream

json.dump(s) and json.load(s)

Python Object Serialization - pickle and json

Python Object Serialization - yaml and json

Priority queue and heap queue data structure

Graph data structure

Dijkstra's shortest path algorithm

Prim's spanning tree algorithm

Closure

Functional programming in Python

Remote running a local file using ssh

SQLite 3 - A. Connecting to DB, create/drop table, and insert data into a table

SQLite 3 - B. Selecting, updating and deleting data

MongoDB with PyMongo I - Installing MongoDB ...

Python HTTP Web Services - urllib, httplib2

Web scraping with Selenium for checking domain availability

REST API : Http Requests for Humans with Flask

Blog app with Tornado

Multithreading ...

Python Network Programming I - Basic Server / Client : A Basics

Python Network Programming I - Basic Server / Client : B File Transfer

Python Network Programming II - Chat Server / Client

Python Network Programming III - Echo Server using socketserver network framework

Python Network Programming IV - Asynchronous Request Handling : ThreadingMixIn and ForkingMixIn

Python Coding Questions I

Python Coding Questions II

Python Coding Questions III

Python Coding Questions IV

Python Coding Questions V

Python Coding Questions VI

Python Coding Questions VII

Python Coding Questions VIII

Python Coding Questions IX

Python Coding Questions X

Image processing with Python image library Pillow

Python and C++ with SIP

PyDev with Eclipse

Matplotlib

Redis with Python

NumPy array basics A

NumPy Matrix and Linear Algebra

Pandas with NumPy and Matplotlib

Celluar Automata

Batch gradient descent algorithm

Longest Common Substring Algorithm

Python Unit Test - TDD using unittest.TestCase class

Simple tool - Google page ranking by keywords

Google App Hello World

Google App webapp2 and WSGI

Uploading Google App Hello World

Python 2 vs Python 3

virtualenv and virtualenvwrapper

Uploading a big file to AWS S3 using boto module

Scheduled stopping and starting an AWS instance

Cloudera CDH5 - Scheduled stopping and starting services

Removing Cloud Files - Rackspace API with curl and subprocess

Checking if a process is running/hanging and stop/run a scheduled task on Windows

Apache Spark 1.3 with PySpark (Spark Python API) Shell

Apache Spark 1.2 Streaming

bottle 0.12.7 - Fast and simple WSGI-micro framework for small web-applications ...

Flask app with Apache WSGI on Ubuntu14/CentOS7 ...

Selenium WebDriver

Fabric - streamlining the use of SSH for application deployment

Ansible Quick Preview - Setting up web servers with Nginx, configure enviroments, and deploy an App

Neural Networks with backpropagation for XOR using one hidden layer

NLP - NLTK (Natural Language Toolkit) ...

RabbitMQ(Message broker server) and Celery(Task queue) ...

OpenCV3 and Matplotlib ...

Simple tool - Concatenating slides using FFmpeg ...

iPython - Signal Processing with NumPy

iPython and Jupyter - Install Jupyter, iPython Notebook, drawing with Matplotlib, and publishing it to Github

iPython and Jupyter Notebook with Embedded D3.js

Downloading YouTube videos using youtube-dl embedded with Python

Machine Learning : scikit-learn ...

Django 1.6/1.8 Web Framework ...


Sponsor Open Source development activities and free contents for everyone.

Thank you.

- K Hong






OpenCV 3 image and video processing with Python



OpenCV 3 with Python

Image - OpenCV BGR : Matplotlib RGB

Basic image operations - pixel access

iPython - Signal Processing with NumPy

Signal Processing with NumPy I - FFT and DFT for sine, square waves, unitpulse, and random signal

Signal Processing with NumPy II - Image Fourier Transform : FFT & DFT

Inverse Fourier Transform of an Image with low pass filter: cv2.idft()

Image Histogram

Video Capture and Switching colorspaces - RGB / HSV

Adaptive Thresholding - Otsu's clustering-based image thresholding

Edge Detection - Sobel and Laplacian Kernels

Canny Edge Detection

Hough Transform - Circles

Watershed Algorithm : Marker-based Segmentation I

Watershed Algorithm : Marker-based Segmentation II

Image noise reduction : Non-local Means denoising algorithm

Image object detection : Face detection using Haar Cascade Classifiers

Image segmentation - Foreground extraction Grabcut algorithm based on graph cuts

Image Reconstruction - Inpainting (Interpolation) - Fast Marching Methods

Video : Mean shift object tracking

Machine Learning : Clustering - K-Means clustering I

Machine Learning : Clustering - K-Means clustering II

Machine Learning : Classification - k-nearest neighbors (k-NN) algorithm




Machine Learning with scikit-learn



scikit-learn installation

scikit-learn : Features and feature extraction - iris dataset

scikit-learn : Machine Learning Quick Preview

scikit-learn : Data Preprocessing I - Missing / Categorical data

scikit-learn : Data Preprocessing II - Partitioning a dataset / Feature scaling / Feature Selection / Regularization

scikit-learn : Data Preprocessing III - Dimensionality reduction vis Sequential feature selection / Assessing feature importance via random forests

Data Compression via Dimensionality Reduction I - Principal component analysis (PCA)

scikit-learn : Data Compression via Dimensionality Reduction II - Linear Discriminant Analysis (LDA)

scikit-learn : Data Compression via Dimensionality Reduction III - Nonlinear mappings via kernel principal component (KPCA) analysis

scikit-learn : Logistic Regression, Overfitting & regularization

scikit-learn : Supervised Learning & Unsupervised Learning - e.g. Unsupervised PCA dimensionality reduction with iris dataset

scikit-learn : Unsupervised_Learning - KMeans clustering with iris dataset

scikit-learn : Linearly Separable Data - Linear Model & (Gaussian) radial basis function kernel (RBF kernel)

scikit-learn : Decision Tree Learning I - Entropy, Gini, and Information Gain

scikit-learn : Decision Tree Learning II - Constructing the Decision Tree

scikit-learn : Random Decision Forests Classification

scikit-learn : Support Vector Machines (SVM)

scikit-learn : Support Vector Machines (SVM) II

Flask with Embedded Machine Learning I : Serializing with pickle and DB setup

Flask with Embedded Machine Learning II : Basic Flask App

Flask with Embedded Machine Learning III : Embedding Classifier

Flask with Embedded Machine Learning IV : Deploy

Flask with Embedded Machine Learning V : Updating the classifier

scikit-learn : Sample of a spam comment filter using SVM - classifying a good one or a bad one




Machine learning algorithms and concepts

Batch gradient descent algorithm

Single Layer Neural Network - Perceptron model on the Iris dataset using Heaviside step activation function

Batch gradient descent versus stochastic gradient descent

Single Layer Neural Network - Adaptive Linear Neuron using linear (identity) activation function with batch gradient descent method

Single Layer Neural Network : Adaptive Linear Neuron using linear (identity) activation function with stochastic gradient descent (SGD)

Logistic Regression

VC (Vapnik-Chervonenkis) Dimension and Shatter

Bias-variance tradeoff

Maximum Likelihood Estimation (MLE)

Neural Networks with backpropagation for XOR using one hidden layer

minHash

tf-idf weight

Natural Language Processing (NLP): Sentiment Analysis I (IMDb & bag-of-words)

Natural Language Processing (NLP): Sentiment Analysis II (tokenization, stemming, and stop words)

Natural Language Processing (NLP): Sentiment Analysis III (training & cross validation)

Natural Language Processing (NLP): Sentiment Analysis IV (out-of-core)

Locality-Sensitive Hashing (LSH) using Cosine Distance (Cosine Similarity)




Artificial Neural Networks (ANN)

[Note] Sources are available at Github - Jupyter notebook files

1. Introduction

2. Forward Propagation

3. Gradient Descent

4. Backpropagation of Errors

5. Checking gradient

6. Training via BFGS

7. Overfitting & Regularization

8. Deep Learning I : Image Recognition (Image uploading)

9. Deep Learning II : Image Recognition (Image classification)

10 - Deep Learning III : Deep Learning III : Theano, TensorFlow, and Keras









Contact

BogoToBogo
contactus@bogotobogo.com

Follow Bogotobogo

About Us

contactus@bogotobogo.com

YouTubeMy YouTube channel
Pacific Ave, San Francisco, CA 94115

Pacific Ave, San Francisco, CA 94115

Copyright © 2024, bogotobogo
Design: Web Master