I spent a month learning how to start coding. Although sometimes challenging, it is a very useful experience.

The most difficult challenge is to choose what to learn and learn it. There are so many languages, frameworks and libraries - and many conflicting suggestions.

Also, some learning resources are much better than others. If I knew which resources to focus on, my progress might be faster.

With this in mind, I think I will share something useful with you. If I feel a lot, I am very lucky. I can spend several hours a day studying these materials.

How fast you do it doesn't reflect your potential as a developer, so don't worry if it takes you longer to complete these materials. However, if you have the time, you can finish it in half a month.

These are the paths I have taken, which does not mean you have to go through them all.

Part I: HTML,CSS and JavaScript

These are the three core languages of all web applications, and it is (almost) impossible to build anything on the web without using them.

HTML

If you have never done any coding before, HTML is a good place to start. It is the core building block of the web. Unlike most other languages, it gives learners real-time visual feedback about the code they are writing.

CSS

CSS is HTML's natural companion. On the surface, it is very simple, although there are some concepts - particularly relating to layout and positioning - that can take some time to understand.

The most difficult part of CSS is understanding the layout. Users asked me most of my layout questions on StackOverflow, so the quick Google search solved most of my problems.

H5 and CSS took me about two days. Usually, when you want to know something about coding, it is always the quickest way to get started directly.

JavaScript

HTML and CSS are not fully mature programming languages, so you will not be able to fully understand programming before you begin to learn programming languages. JavaScript is the programming language of the Web.

To be honest, although I tried js programming in junior high school, I can't even count the real Coding. Until now, I am also JavaScript Cainiao, so I can't give some help and suggestions on it.

since I started to learn backend, js only took me a short period of time.

GO

Go is an extremely cool language, especially in modern efficient Web programming.

Go is the language that I choose to learn from the backend, and it is also the language that I prefer to learn more. As a beginner in the language, I still can't distinguish the characteristics and quality of various languages, which usually requires years of coding experience or experience of using multiple languages to be clear.

I have experienced the use of C ++ STL and certain Java coding, but from the perspective of the past, it can only be achieved in more than ten days.

I spent about three days on The Way To Go. Before that, I read The guide for getting started with Go in The Cainiao tutorial. Although this book is relatively brief about the advanced features of the Go language, it is undoubtedly enough as an introduction.

Even if I read English, maybe you prefer reading Chinese books, but as a coding staff, English is undoubtedly the language that must be learned. after that, I did a little crawler practice, involving goquery,gocolly crawled Feilu's novel website first and then crawled various business information of various regions.

This probably took me a day. (Knowledge of reading documents learned from here)

Then I started reading the Go Source Code package, traversing each function of each package through their godoc. When I started reading the internal logic of each function, I was stopped because of the huge knowledge system. It took me about three to four days. (Drawing skills learned from here)

At this time, I started to develop a cloud API access service for sending text messages. At that time, understanding and accessing the Alibaba Cloud market brought me trouble, and it was never easy to find a solution to the demand. It took about a week to learn about the access between go and MySQL and the HTTP communication between different terminals.

The following is the background management of the Carousel graph. Here, I began to learn about table design and learn more about MySQL.

Finish the carousel graph (the more you consider, the more you need to do, but the coding may not be changed, it should follow the power law) and start the design of the order system, which is the most difficult part to learn the background functions.

At the same time, I also spent a lot of time consulting and designing, and also contacted with the use of gorm. After finishing the design of the order system, I got a great improvement because I considered a lot of adaptation to the actual situation. After that, the more difficult business must be the game server.

After that, I completed the background construction of an app, which was combined and spliced by various modules. The difficulty here is mainly in teamwork, so it is not detailed here.

here, my go-to-study process has come to an end.

Part 2: other essential parts of any developer toolkit

Git and GitHub

Git is the so-called versioning system, and GitHub is the most popular cloud-based Git Service.

In short, Git allows you to back up files, track changes, share your progress with other developers, and collaborate with projects.

When you have just learned and do not need to use Git, it can feel like a hard job. However, this is worthwhile: most professional developers not only rely on GitHub (or similar services) but also have active GitHub configuration files to help you ensure worker safety.

The earlier you start to hand over your work to GitHub, the more experience you will have as a developer.

Command line

the command line is the most important part. It is necessary to understand the operation of the command line. At the same time, it is very simple to learn. There are not only many tutorials on the Internet, but also many private Linux dishes of brother no.

However, my suggestion for tools is that if it is not necessary, I just need to know.

The best way to use Git is through "terminal" or "command line", which is also important for modern Web developers, allowing you to use NPM (Node package manager) or Yarn you can easily add third-party packages to your website. There are also many websites that can help you take this step

part 3: What is the next step?

This is all I have learned in a month.

The content to be learned includes Kubernetes,Istio, and gRPC. This is a lot of technology, and it is best to know them too!

• available per second Limit put tokens into the bucket, or, every time 1/Limit add a token to the second bucket
• maximum storage in buckets burst tokens. If the bucket is full, the new token will be discarded.
• when an N is consumed when the data packet of the unit arrives N tokens, and then send the packet
• if the available token in the bucket is less than N, the packet will be cached or discarded

token bucket algorithm

the token bucket algorithm is the most commonly used algorithm in network Traffic Shaping (Traffic Shaping) and Rate Limiting (Rate Limiting).

Typically, the token bucket algorithm is used to control the number of data sent to the network and allow the sending of burst data.

overview

this package is based on the Token Bucket algorithm (Token Bucket) to implement throttling, which is very easy to use. RateLimiter is often used to limit the access rate to some physical or logical resources. It supports three methods,

• AllowN() If you can't get it, return it immediately.
• WaitN() It is temporarily lined up. When the token is sufficient, it may be returned to the position because of the Cancel of Context.
• ReserveN() Started directly, but the predecessors dug the pit and filled it. The next request will pay the price for this, and wait until the tokens will make up for the air. There is enough token in the barrel.

Working instance

assume that one is working RateLimiter

allow and wait

For a Ratelimiter that generates a token per second, every second without a token, we will add a token 1.

If the Ratelimiter does not use it in 10 seconds, then tokens become 10.0. At this time, a request arrives and requests three tokens, we will serve it from the token in Ratelimiter, tokens to 7.0. After this request, another request comes and requests 10 tokens.

We will from the remaining 7 token cards from RatelimiterFor this request, there are three tokens left, we will get them from the new token produced by Ratelimiter.

We already know that the Ratelimiter produces 1 new token per second, which means that the above request still requires the three commands required for above request. The card requires it to wait for 3 seconds.

reserve

Imagine a Ratelimiter generated a token per second, and now it is not used (in the initial state). If an expensive request requires 100 token cards. If we choose to let this request wait for 100 seconds before allowing it to execute, this is obviously ridiculous.

Why do we do nothing but just wait for 100 seconds? A better approach is to allow this request to execute immediately (no different from all), and then postpone the subsequent request to the right time point.

We allow this expensive task to perform immediately and delay the subsequent request for 100 seconds. This strategy is to let the task execute and wait at the same time.

About timetoact

An important conclusion: Ratelimit does not remember the last request, but the next request allows the time to execute. This can also tell us very straightforwardly that the time interval of reaching the next scheduling time point.

The Ratelimiter is also very simple: the next scheduling time has passed. The difference between this time and the current time is how long the Ratelimiter has not been used. We will translate this time into tokens.Limit == 1), and just one request per second, then tokens will not grow.

burst

Ratelimiter has a barrel capacity that is directly discarded when the request is greater than the capacity of this barrel.

https://github.com/golang/time/blob/master/rate/rate.go

this article describes how to analyze and design State persistence management through React Hooks.

Analysis

Normal frontend, components are class files, maintain their own status, and are not easy to reuse.

First, separate the UI and status of the component and connect them with Action, as shown in the following figure.
UI=f(State).png
Action is an operator

image.png

According to the Hooks, useState() method Object.is comparison algorithm

To compare the state. And useEffect() then provide choose to let it when only some values change

Design

virtual part Mathematical model

the real number is not complete, and the imaginary part is introduced.

Imaginary number, you only need to remove the imaginary part to represent the real number.

The same is true for Curry Func.

Similarly: f(S,∆) medium f(S) represents a real number, which is incomplete. You can add the preceding statement to indicate all the situations.

The changed dimension changes from a one-dimensional line to a two-dimensional plane.

In addition, the framework uses f(S, ∆) it is still a one-dimensional line, but it is actually any line of the plane because f(S,...) has already been determined by the user, that is, a plane dimension reduction is selected in multiple dimensions in the code.

Persistence

you need to store data in one place, such as local,session, and remote.

Connector

how does a component distribute triggered events to the State for processing? Communication is required. Due to the js single-thread model, the shared memory design is selected. Add a Connector communication. The Component Component is how to notify the State of changes. Shared memory, using Connector middle layer.

Action by CurryFunc

how do I know which states have been changed by the framework user-defined Action? That is, the specific value of azone is unknown. Use Curry Func to meet the requirement of delay evaluation. Use fg(S){return f(∆) } replace f(S,∆) use the State framework by users themselves f(∆) register your own state change operator. State framework for developers fg(S) , just pass in all the State. Due to the existence of the React Hooks, the state is used. f(S,∆) the function to update. Therefore, the framework leaves useState() interface and returns f(∆)

For user status management.

Redux is also based on this function model, which has been officially used in Hooks. useReducer(reducer, initialState) it is implemented. The reducer is set f(S,∆) , and it returns state and dispatch, where state is S A and dispatch is f(∆)

Redux

Redux is also based on this function model, which has been officially used in Hooks. useReducer(reducer, initialState) it is implemented. The reducer is set f(S,∆) , and it returns state and dispatch, where state is S A and dispatch is f(∆)

function useReducer(reducer, initialState) {
  const [state, setState] = useState(initialState);
  function dispatch(action) {
    const nextState = reducer(state, action);
    setState(nextState);
  }
  return [state, dispatch];
}

in our view, it also implements Connector internally.

Implementation

Persistence

the first is to implement storage through Hooks, using Local Store

function useLocalJSONStore(key, defaultValue) {
    const [state, setState] = useState(
      () => JSON.parse(localStorage.getItem(key)) || defaultValue
    );
    useEffect(() => {
      localStorage.setItem(key, JSON.stringify(state));
    }, [key, state]);
    return [state, setState];
}

provides persistent storage and external state management support. Considering that we will use Go as the front end:

1. use Hooks and sqlite3 to store locally
2. use Hooks to communicate with Go

Connector

to implement global status notifications using Hooks. First understand useState() to access to setState() will trigger the current component rendering: https://zh-hans.reactjs.org/docs/hooks-state.html Connector let using global state components subscription to connect on the global status updates, will own setState()Pass in the update queue when any of the components use dispatch()  import { useEffect } from "react"

import { useEffect } from "react"

const Connector = {}

const Broadcast = (name, state) => {
    if (!Connector[name]) return;
    Connector[name].forEach(setter => setter(state))
}

const Subscribe = (name, setter) => {
    if (!Connector[name]) Connector[name] =[];
    Connector[name].push(setter)
}

const UnSubscribe = (name, setter) => {
    if (!Connector[name]) return
    const index = Connector[name].indexOf(setter)
    if (index !== -1) Connector[name].splice(index, 1)
}

const connect = (name,setState) => {
    console.log('connect')
    useEffect(() =>{
        Subscribe(name, setState)
        console.log('subscirbe',name)
        return () => {
            UnSubscribe(name,setState)
            console.log('unsubscribe',name)
        }
    },[])
}

useStore

user usage useStore() to obtain the global status and dispatch() function. The internal implementation is to State Hook and get setState() register to the subscription list.

import {Broadcast,connect} from './Connector'
import {useState} from 'react'

export function useStore(key,value) {
    const [state,setState] = useState(value)
    connect(key,setState)

    return [state, (key,value) => {
        Broadcast(key,value)
    }]
}

Current status

Use

Use useStore(key, value) OK.

import {useStore} from './useStore'

export function Counter({key,initialCount}) {
    // const [count, setCount] = useLocalJSONStore(keyname, initialCount);
    const [state, dispatch] = useStore(key,initialCount)
    return (
      <>
        Count: {state}
        <button onClick={() => dispatch(keyname,initialCount)}>Reset</button>
        <button onClick={() => dispatch(keyname,state-1)}>-</button>
        <button onClick={() => dispatch(keyname,state+1)}>+</button>
      </>
    );
  }

Advanced

• [ ] asynchronous status
• [ ] Decorator

First of all, in practical operation, it can be seen that skilled people are inferior to experienced people in all aspects, but they usually do better than inexperienced people. By many views gained from the experience of the universal Judgment (Generic), we believe that technology was born.

Generally, we think that people with technology have a deeper understanding of this kind of thing, (in short, smarter) because their behaviors will be guided, and their starting point is reason rather than inertia. We also believe that people with technology master the ability to impart theories, but experienced people cannot teach others.

Do you want to judge whether you are good at a certain technology and a master in this field?

when you think that you can do something if others ask you but you can only tell the specific situation and specific practices, you are not a wise professor. If the universal judgment you say is not considered dialectically by yourself, you are not a Master of this event.

According to this statement, the concept of a Technologist is close to a wise man, and wisdom is also explained as knowledge about the origin and principle of things. Readers can now think about what characteristics people with high IQ have in the concept we are talking about?

from the above ideas, it is not surprising that mathematicians call them philosophers or wise men, because this discipline is based on the basic principles (Root) rather than a series of secondary disciplines.

Therefore, the more common the principle is, the more it is regarded as truth. People who think they have mastered the truth tend to be frustrated in the field of new knowledge.

Since the opinions that can be collected correspond to the infinite things, in reality, I am inconvenient to think that the universal principle of a kind of things also lacks the existence of truth because there are infinite kinds.

So when experts claim to be masters in other fields, I don't fear to think that they are not very good in this field.

The above metaphysical discussion is not to explore whether the truth exists in a pessimistic way. Instead, I want to express my praise for practical operation from the perspective of reality and the method of getting the so-called twice the result with half the effort by deeply learning the principles of things.

On the other hand, as programmers, top programmers, and technicians' technologies, they can explain the measurement of the difference in value they create.

It is not difficult to become an expert in a certain field through practice but based on the viewpoint of experience summary, it will be our goal to put forward universal fragments (.e. creation, design, and the invention of new technologies).

Refer

• Docstore · Go CDK
• urls.go - google/go-cloud - Sourcegraph
• driver.go - Go

Design objectives:

through the abstraction layer, we can mask differences, provide services in a standardized way, and configure business applications through description files.

Provides design ideas and guidelines for applications that use document storage.

Intro

common in MongoDB document Storage provides an abstraction layer.

Document Storage is a service that stores data in semi-structured JSON-like documents. These documents are grouped into collections. Like other NoSQL databases, document storage is modeless.

The design needs to support adding, retrieving, modifying, and deleting documents. docstore Driver implementation of various services, including cloud and local solutions. You can develop applications locally and then reconfigure them to multiple cloud providers with minimal initialization.

设计

Structuring Portable Code

Structuring Portable Code the non-interface design imitates the database/SQL package of golang and wraps the existing common logic into the structure. The internal fields of the structure are driver interfaces. The method provided externally is the method corresponding to the structure rather than the implementation of the direction provided driver.

The advantage of this design is that there is no need to implement general logic processing for each interface, and the code can be transplanted. In some cases, you only need to add and modify methods on the structure and do not need to destroy the method design in the interface. You can also mask some assertion logic. When switching different drivers, users do not need to determine the implementation of some optional interfaces.

Structuring Portable Code · Go CDK [sql package - database/sql - Go Packages](https://pkg.go.dev/database/sql#DB

Code like below：

// Define
type DB interface{
    Exec(sql string)error
}

// Realize
type mysql struct {}
func (m *mysql) Exec(sql string) error {return nil}

// Execute
sql.Database.Exec("")

// package and structure
package sql

type DB struct {
    driver driver.DB
}

// higher level logic
func (db *DB) AnySignature(anyParams string) (anyReturn error) {
    //... 
    db.driver.Exec("...")
    //...
    return nil
}
// Define
package driver 

type DB interface{
    Exec(sql string)error
}

// Realize
type mysql struct {}
func (m *mysql) Exec(sql string) error {return nil}

// Execute
sql.DB.AnySignature("")

Actions List

For MongoDB, batch processing can be carried out to improve efficiency. As the shielding layer of packaging, we hope to obtain this benefit according to the actual processing of driver. A queue or cache is required to submit a batch operation.

Batch write operations-MongoDB-CN-Manual

• [x] I think it is enough to undertake Google Go CDK design

Driver Map & Opener

Inherited from the Mysql Driver registration method, through the golang standard import_" github.com/xxx/driver" different database drivers can be introduced. The principle is to use a global Map.

Go CDK has upgraded the Opener feature. The original custom URL Parsing method is "mysql", "user:password@/dbname" the features of the new version are blob+file:///dir even + + prefix (e.g. blob+bucket+file:///dir) for Google Cloud SDK, the same URL can provide different functions. However, in our opinion, this function does not have much effect for the time being, so we will block their design.

Dependency Injection wire

Go CDK use the wire project to inject dependencies to automatically switch the structure of different backend providers to the SDK. Different from the way Dapr accesses different services, Dapr uses the yaml description to determine the different plug-ins that are enabled.

For example, you need wire.Build() indicates the new function of the driver to be introduced.

It has little impact on this project and may not be added for the time being.

UUID usage

mongoDB, each entry must have a Key, which can be passed through parameters.

type Player struct {
    ID   interface{} `docstore:"_id,omitempty"`
    Name string
}

The simplest is to indicate the_id field directly in the structure.

docstore.OpenCollection(context.Background(), "mongo://my-db/my-coll?id_field=name")

You can also use the URL Parameter in the high-level abstraction. The following example specifies the_ID as the name field of the above-mentioned high-level abstraction, which is used by the underlying layer mongodocstore.OpenCollection mapping relationship, will automatically generate mongo official driver type primitivie.ObjectID

coll, err := mongodocstore.OpenCollection(mcoll, "id", nil)

type IDer struct {
    ID primitive.ObjectID
}

you can also use mongodocstore.OpenCollectionWithIDFunc to specify how to generate an ID.

nameFromDocument := func(doc docstore.Document) interface{} {
    return primitive.NewObjectID()
}
coll, err := mongodocstore.OpenCollectionWithIDFunc(mcoll, nameFromDocument, nil)

Summary

We have completed the access design and understanding of Document Store and can perform basic operations on adding, deleting, modifying, and querying docstores. Next, we will build service applications based on this layer of abstraction.

For special functions of different docstores, you can add them to docstore to determine whether they are target-driven and change the method of external exposure.

function

the following shows the functions of the library.

Connect MongoDB

The default mongo driver uses MONGO_SERVER_URL link to the server, so you can use code to set it here or directly set it by using environment variables.

the following meaning is from mongodb://localhost:27017 the link on the server is called my-dbin the database my-coll document. The unique field name of mongo is name.

os.Setenv("MONGO_SERVER_URL", "mongodb://localhost:27017")

coll, err := docstore.OpenCollection(context.Background(), "mongo://my-db/my-coll?id_field=name")
defer coll.Close()

Corresponding display structure

type Player struct {
    Name             string `docstore:"name,omitempty"`
    Score            int
    DocstoreRevision interface{}
}

Create

coll.Create(ctx, &Player{Name: "Pat", Score: 7});

Get

coll.Get(ctx, &Player{Name: "Pat"});

Queries

you may need to manually create indexes to complete the query function.

import (
    "context"
    "fmt"
    "io"

    "gocloud.dev/docstore"
)

// Ask for all players with scores at least 20.
iter := coll.Query().Where("Score", ">=", 20).OrderBy("Score", docstore.Descending).Get(ctx)
defer iter.Stop()

// Query.Get returns an iterator. Call Next on it until io.EOF.
for {
    var p Player
    err := iter.Next(ctx, &p)
    if err == io.EOF {
        break
    } else if err != nil {
        return err
    } else {
        fmt.Printf("%s: %d\n", p.Name, p.Score)
    }
}

Update a single field of an Update entry

pat2 := &Player{Name: "Pat"}
err := coll.Actions().Update(pat, docstore.Mods{"Score": 15}).Get(pat2).Do(ctx)

Replace

completely replace the entire entry

coll.Replace(ctx, &Player{Name: "Pat", Score: 15})

Put

the Put function is equivalent to CreateOrUpdate

coll.Put(ctx, &Player{Name: "Pat", Score: 15})

Delete

coll.Delete(ctx, &Player{Name: "Pat", Score: 15})

More examples

• CLI Sample
• Order Processor sample
• docstore package examples

before we begin, let's explore the differences between HTTP and RPC. The reason why gRPC is discussed here is that no one uses common RPC.

HTTP is a common communication method used for business coding, and its popularity is needless to say. As a Web programmer, HTTP Server programming is its core skill. RPC is also indispensable in microservices.

Can programmers who are familiar with one of the encoding quickly get started with the business coding of another transmission method?

After all, the business logic is consistent, which seems to be only different in network transmission.

Colleagues who have done this know that the differences in business coding are not small. Although the differences are constrained to the transport layer through the abstraction layer during design, there is no framework to block the differences in implementation. Therefore, coding students need to go deep into it and handle it by themselves.

For example, you need to learn more about envoy and proto files, how to encode requests and return values, and how to use specific protobuf to parse message packets in your business.

The differences can be shielded at the abstraction level. We still need to write detailed differences in implementation. These are the operations that some programmers can replace with frameworks.

Top programmers and beginners, beginners, and idiots

the role of the framework is to make correct coding behavior without thinking.

There are enough ecological libraries for the current language to help compile various coding types. When lacking, some ecosystems can be transplanted by referring to other languages to reduce the workload. However, not every programmer can do such behavior at any time.

Google is a friend. Business Code they often get into trouble because of something they don't know so that no matter how their skills or intelligence are, they can't solve the problem.

In the business, some coding work will be compared to moving bricks. Programmers are described as manual work to move code from here to there. However, when someone participates in the process, the error probability will also be related to the state of a person.

Through Murphy's law, we can recognize that errors must occur in these processes. How do reduce personal decisions to ensure high quality and high output of assembly line coding manpower?

If you want to treat yourself as an idiot coder and leave the error-prone parts to tools, the framework will generate great benefits.

Nothing is built on Stone; Everything is built on sand, but we must build sand as stone. -Jorge Louis Borges

the following are some examples:

• code review:

architects not only need to formulate process standards, but also need to supervise the implementation. Code review is the major part of the workload. However, there are thousands of people, and code writers have their ideas. There may even be a design-based cohesion function, which is scattered at all levels in implementation, and the review process is even more inefficient.

Constraints can be carried out through the framework, which is also the wisdom of software engineering. By increasing restrictions, standards can be formulated to provide efficiency.

• Best Practices:

business code usually uses simple addition, deletion, modification, query data, and target resources. At the same time, there are some common functional requirements, such as JWT.

The framework can shield these differences. For example, JWT only has different types of tokens carried by HTTP, and ORM shields the actual data storage software interfaces in the background for addition, deletion, query, modification, and modification.

This is another wisdom of computer science, solving problems by adding a middle layer. Framework users can switch to different implementations without thinking.

If the best practices provided by the framework cannot meet the requirements, it is time for the document to show its role. Technical personnel-oriented documentation is useful only when problems occur.

The dilemma of microservices caused by abstract hierarchy and abstract leakage

Google software engineering mentions three key differences between programming and software engineering: time, scope, and trade-offs.

However, the idea of the framework is beautiful enough, but the realization, in reality, is full of trade-offs and the pursuit of perfection.

Even if the strange requirements of a specific time limit on the business side are excluded. The design cannot be accomplished overnight and a perfect abstract design can be completed.

Abstract leakage refers to the abstraction of implementation details that should be hidden during software development, which inevitably exposes the underlying details and limitations.

Not to mention that a complete system has more than one or two levels. How to make reasonable abstraction and promote it as a standard is a long-term practice and change in many microservice frameworks and coding fields.

Abstraction means unification, while behind the abstraction level, it usually means the actual services with different characteristics. Do you use the union or intersection of these services for abstraction? Whether to consider extended compatibility or functionality.

For more information, see another article. Mongo Doc access design, is practical experience.

Also The API of Dapr. Many Interfaces of Golang (IO, SQL, and Net) can see abstract practical practices.

For example, designers will struggle with whether to provide a certain function to the outside, so they have done a lot of work to provide it. However, in terms of function usage, it may be a pseudo requirement or a simple shielding. However, in actual scenarios, it is necessary to have a lower layer of functions, and the abstraction level is still broken down.

At this point, everyone understands that it falls into specific scenarios and analyzes specific problems. Therefore, a microservice framework that has passed the postgraduate entrance examination for a long time must have solved many problems in the target scenario.

This reminds me that programmers always pursue new technologies. New microservice frameworks usually have high expectations, hoping that they can completely solve the problems encountered in practice that the old frameworks cannot solve. Finally, expectations often fail. Why can we expect a new untested framework to meet the needs of the technical framework that has been designed and modified many times in practice in specific fields?

Back to our question at the beginning, is there a framework that unifies the HTTP Channel and gRPC Channel, and only needs to write the handler's internal code without paying attention to other work?

In the modern framework, Dapr did accomplish this.

What about the abstract cost?

The field type in the Protobuf is lost, and it is considered a payload. The handler has different self-processing types, which is consistent with HTTP abstraction.

Is it true that such an abstraction layer has just come up with now? If you have a deeper understanding of computer science, you will find that some past ideas shine brilliantly in new scenarios.

Time is the most significant variable (for example, previous programmers needed to deduct bytes. Now, do you still need to care about insufficient memory for personal PC and cloud coding?).

Summary

The above describes the problems related to the microservice framework considered in the experience. Just raising questions is a hooligan. My opinions and suggestions are mentioned a lot in the article.

To make a summary, it is:

providing a fool-like automated microservice framework enables programmers to make fewer decisions and make better decisions.

Only by using the time saved to innovate business links and business models, and not being involved in non-creative work such as environment building, can workers feel the value of innovation and self-achievement.

ReadLink

• pkg/kubelet/nodestatus/setters.go
• /pkg /kubelet /kubelet_node_status.go

Directory Layout

pkg/kubelet/nodestatus
 |- setters.go
 |- setters_test.go

Setter

// Setter modifies the node in-place, and returns an error if the modification failed.
// Setters may partially mutate the node before returning an error.
type Setter func(node *v1.Node) error

the Setter function defines a function that performs operations on the v1.Node object. If an error is returned, the Node object may also be changed. From the function definition, you can see its usage: use functions to generate different setters for a class of modification of Node objects. In this way, you can modify the state of a Node. Use the simplest func GoRuntime() Setter example:

// GoRuntime returns a Setter that sets GOOS and GOARCH on the node.
func GoRuntime() Setter {
    return func(node *v1.Node) error {
        node.Status.NodeInfo.OperatingSystem = goruntime.GOOS
        node.Status.NodeInfo.Architecture = goruntime.GOARCH
        return nil
    }
}

this mode belongs to the middleware operation mode. You can contact middleware for understanding.

Setter List

we learned the setter mode changed by Node status. Currently, the code contains the following 12 setters:

• NodeAddress returns a Setter that updates address-related information on the node.：updates address-related fields, such as IP address and hostname (typically the hostname variable in kubelet).
• MachineInfo returns a Setter that updates machine-related information on the node.：updates fields related to host information, such as the maximum number of pods, the number of pods allocated to each core, and the number of resources.
• VersionInfo returns a Setter that updates version-related information on the node.：containerRuntime version, cadvisor version
• DaemonEndpoints returns a Setter that updates the daemon endpoints on the node.
• Images returns a Setter that updates the images on the node.：updates image information.
• GoRuntime returns a Setter that sets GOOS and GOARCH on the node.：GOOS GOARCH information
• ReadyCondition returns a Setter that updates the v1.NodeReady condition on the node.：determines whether the node is in the Ready state from Kubelet fields such as the error return function in the runtimeState.
• MemoryPressureCondition returns a Setter that updates the v1.NodeMemoryPressure condition on the node.
• PIDPressureCondition returns a Setter that updates the v1.NodePIDPressure condition on the node.
• DiskPressureCondition returns a Setter that updates the v1.NodeDiskPressure condition on the node.
• VolumesInUse returns a Setter that updates the volumes in use on the node.
• VolumeLimits returns a Setter that updates the volume limits on the node.

Setter 的入参通常是 Kubelet 中的字段，自然使用是通过 Kubelet 去初始化使用。

Kubelet Node Status

/pkg /kubelet /kubelet_node_status.go

Setter 使用处

after all setters are initialized in the defaultNodeStatusFuncs function, the function returns a Setter array.

kubelet_node_status.go? L613

// defaultNodeStatusFuncs is a factory that generates the default set of
// setNodeStatus funcs
func (kl *Kubelet) defaultNodeStatusFuncs() []func(*v1.Node) error {
    // if cloud is not nil, we expect the cloud resource sync manager to exist
    var nodeAddressesFunc func() ([]v1.NodeAddress, error)
    if kl.cloud != nil {
        nodeAddressesFunc = kl.cloudResourceSyncManager.NodeAddresses
    }
    var validateHostFunc func() error
    if kl.appArmorValidator != nil {
        validateHostFunc = kl.appArmorValidator.ValidateHost
    }
    var setters []func(n *v1.Node) error
    setters = append(setters,
        nodestatus.NodeAddress(kl.nodeIPs, kl.nodeIPValidator, kl.hostname, kl.hostnameOverridden, kl.externalCloudProvider, kl.cloud, nodeAddressesFunc),
        nodestatus.MachineInfo(string(kl.nodeName), kl.maxPods, kl.podsPerCore, kl.GetCachedMachineInfo, kl.containerManager.GetCapacity,
            kl.containerManager.GetDevicePluginResourceCapacity, kl.containerManager.GetNodeAllocatableReservation, kl.recordEvent),
        nodestatus.VersionInfo(kl.cadvisor.VersionInfo, kl.containerRuntime.Type, kl.containerRuntime.Version),
        nodestatus.DaemonEndpoints(kl.daemonEndpoints),
        nodestatus.Images(kl.nodeStatusMaxImages, kl.imageManager.GetImageList),
        nodestatus.GoRuntime(),
    )
    // Volume limits
    setters = append(setters, nodestatus.VolumeLimits(kl.volumePluginMgr.ListVolumePluginWithLimits))

    setters = append(setters,
        nodestatus.MemoryPressureCondition(kl.clock.Now, kl.evictionManager.IsUnderMemoryPressure, kl.recordNodeStatusEvent),
        nodestatus.DiskPressureCondition(kl.clock.Now, kl.evictionManager.IsUnderDiskPressure, kl.recordNodeStatusEvent),
        nodestatus.PIDPressureCondition(kl.clock.Now, kl.evictionManager.IsUnderPIDPressure, kl.recordNodeStatusEvent),
        nodestatus.ReadyCondition(kl.clock.Now, kl.runtimeState.runtimeErrors, kl.runtimeState.networkErrors, kl.runtimeState.storageErrors, validateHostFunc, kl.containerManager.Status, kl.shutdownManager.ShutdownStatus, kl.recordNodeStatusEvent),
        nodestatus.VolumesInUse(kl.volumeManager.ReconcilerStatesHasBeenSynced, kl.volumeManager.GetVolumesInUse),
        // TODO(mtaufen): I decided not to move this setter for now, since all it does is send an event
        // and record state back to the Kubelet runtime object. In the future, I'd like to isolate
        // these side-effects by decoupling the decisions to send events and partial status recording
        // from the Node setters.
        kl.recordNodeSchedulableEvent,
    )
    return setters
}

The array is assigned to the setNodeStatusFuncs of kubelet.

    // Generating the status funcs should be the last thing we do,
    // since this relies on the rest of the Kubelet having been constructed.
    klet.setNodeStatusFuncs = klet.defaultNodeStatusFuncs()

SyncNodeStatus Procedure

how do Kubelet use these Kubelet? The core is syncNodeStatus functions.

kubelet_node_status

// syncNodeStatus should be called periodically from a goroutine.
// It synchronizes node status to master if there is any change or enough time
// passed from the last sync, registering the kubelet first if necessary.
func (kl *Kubelet) syncNodeStatus() {
    kl.syncNodeStatusMux.Lock()
    defer kl.syncNodeStatusMux.Unlock()

    if kl.kubeClient == nil || kl.heartbeatClient == nil {
        return
    }
    if kl.registerNode {
        // This will exit immediately if it doesn't need to do anything.
        kl.registerWithAPIServer()
    }
    if err := kl.updateNodeStatus(); err != nil {
        klog.ErrorS(err, "Unable to update node status")
    }
}

syncNodeStatus the function is called periodically in goroutine to synchronize the node status to the master.

入口 Entry

currently, it is called in three places:

1. kubelet.go? L1428:26 in the Run function of the Kubelet, start goroutine for periodic synchronization.

   go wait.JitterUntil(kl.syncNodeStatus, kl.nodeStatusUpdateFrequency, 0.04, true, wait.NeverStop)
   

2. kubelet.go? L2433:7 : performs one-time synchronization in the fastStatusUpdateOnce function.

   func (kl *Kubelet) fastStatusUpdateOnce() {
    for {
        ...
        kl.syncNodeStatus()
        return
    }
   }
   

3. nodeshutdown_manager_linux.go? L283:11 : it is called in the start() function of nodeshutdownmanager, which is actually a goroutine and is triggered only after the shutdown event is received from the channel.

   if isShuttingDown {
    // Update node status and ready condition
    go m.syncNodeStatus()
   
    m.processShutdownEvent()
   }
   

注册 RegisterWithAPIserver

if kubelet needs to be registered, a for loop is executed to wait for registration to the APIServer.

for {
    time.Sleep(step)
    step = step * 2
    if step >= 7*time.Second {
        step = 7 * time.Second
    }

    // 1. 获取 node 对象及其信息
    node, err := kl.initialNode(context.TODO())
    if err != nil {
        klog.ErrorS(err, "Unable to construct v1.Node object for kubelet")
        continue
    }

    klog.InfoS("Attempting to register node", "node", klog.KObj(node))
    // 2. 注册到 APIServer 中去
    registered := kl.tryRegisterWithAPIServer(node)
    if registered {
        klog.InfoS("Successfully registered node", "node", klog.KObj(node))
        kl.registrationCompleted = true
        return
    }
}

1. node, err := kl.initialNode(context.TODO()) : obtains the node object and its information.
2. registered := kl.tryRegisterWithAPIServer(node) : Register to APIServer

Use Setter

Function tryUpdateNodeStatus (kubelet_node_status.go? L470:20) : the processing part of the volumeManager is omitted.

// tryUpdateNodeStatus tries to update node status to master if there is any
// change or enough time passed from the last sync.
func (kl *Kubelet) tryUpdateNodeStatus(tryNumber int) error {
    originalNode := node.DeepCopy()
    ...
    kl.setNodeStatus(node)
    ...
    // Patch the current status on the API server
    updatedNode, _, err := nodeutil.PatchNodeStatus(kl.heartbeatClient.CoreV1(), types.NodeName(kl.nodeName), originalNode, node)
    ...
    return nil
}

kl.setNodeStatus just traverses all the Setter functions we mentioned just now.

func (kl *Kubelet) setNodeStatus(node *v1.Node) {
    for i, f := range kl.setNodeStatusFuncs {
        klog.V(5).InfoS("Setting node status condition code", "position", i, "node", klog.KObj(node))
        if err := f(node); err != nil {
            klog.ErrorS(err, "Failed to set some node status fields", "node", klog.KObj(node))
        }
    }
}

Conclusion

we have learned:

1. what are the change functions of the Node Status and what rules are followed to sign the function.
2. how to register a Setter function to a Kubelet.
3. Kubelet when these setters are called to change the status of a Node.

Next:

1. you can try to add a custom setter function.
2. Kubernetes code is not as neat as the design. Some todo can be changed after reading this article and code. Try to decouple the code. (You can also find it by searching todo in the code.)

ReadLink

• configmap manager
• pkg/kubelet/secret/secret_manager.go

Configmap Manager

// Manager interface provides methods for Kubelet to manage ConfigMap.
type Manager interface {
    // Get configmap by configmap namespace and name.
    GetConfigMap(namespace, name string) (*v1.ConfigMap, error)

    // WARNING: Register/UnregisterPod functions should be efficient,
    // i.e. should not block on network operations.

    // RegisterPod registers all configmaps from a given pod.
    RegisterPod(pod *v1.Pod)

    // UnregisterPod unregisters configmaps from a given pod that are not
    // used by any other registered pod.
    UnregisterPod(pod *v1.Pod)
}

接口非常简单。

1. GetConfigMap ： 通过 namespace 和 name 获取对应 ConfigMap 对象。
2. RegisterPod(pod *v1.Pod)：把指定 Pod 对象 yaml 指定的 configmap 注册到 Controller 中管理
3. UnregisterPod(pod *v1.Pod)：把指定 Pod 对象 yaml 指定的 configmap 从 Controller 中注册管理中删除，注意 ConfigMap 需要没有任何其他已注册的 Pod 引用（即无被依赖项）才可以删除

当前代码中有两种 manager 的实现

-NewCachingConfigMapManager(kubeClient clientset.Interface, getTTL manager.GetObjectTTLFunc) Manager：该实现有两点逻辑

• 当一个 Pod 创建或者更新时，所有的 configmap 缓存都失效。
• GetObject() 调用首先从本地缓存获取，失败则访问 APISever 并刷新 configmap 的缓存。

// NewCachingConfigMapManager creates a manager that keeps a cache of all configmaps
// necessary for registered pods.
// It implement the following logic:
// - whenever a pod is create or updated, the cached versions of all configmaps
//   are invalidated
// - every GetObject() call tries to fetch the value from local cache; if it is
//   not there, invalidated or too old, we fetch it from apiserver and refresh the
//   value in cache; otherwise it is just fetched from cache
func NewCachingConfigMapManager(kubeClient clientset.Interface, getTTL manager.GetObjectTTLFunc) Manager {
    getConfigMap := func(namespace, name string, opts metav1.GetOptions) (runtime.Object, error) {
        return kubeClient.CoreV1().ConfigMaps(namespace).Get(context.TODO(), name, opts)
    }
    configMapStore := manager.NewObjectStore(getConfigMap, clock.RealClock{}, getTTL, defaultTTL)
    return &configMapManager{
        manager: manager.NewCacheBasedManager(configMapStore, getConfigMapNames),
    }
}

• NewWatchingConfigMapManager(kubeClient clientset.Interface, resyncInterval time.Duration) Manager：
  • 当一个 Pod 创建或者更新时，会对指定该 Pod 引用的资源，并且该资源未被其他 Pod 引用进行独立的 watch。
  • GetObject() 调用首先从本地缓存获取

// NewWatchingConfigMapManager creates a manager that keeps a cache of all configmaps
// necessary for registered pods.
// It implements the following logic:
// - whenever a pod is created or updated, we start individual watches for all
//   referenced objects that aren't referenced from other registered pods
// - every GetObject() returns a value from local cache propagated via watches
func NewWatchingConfigMapManager(kubeClient clientset.Interface, resyncInterval time.Duration) Manager {
    listConfigMap := func(namespace string, opts metav1.ListOptions) (runtime.Object, error) {
        return kubeClient.CoreV1().ConfigMaps(namespace).List(context.TODO(), opts)
    }
    watchConfigMap := func(namespace string, opts metav1.ListOptions) (watch.Interface, error) {
        return kubeClient.CoreV1().ConfigMaps(namespace).Watch(context.TODO(), opts)
    }
    newConfigMap := func() runtime.Object {
        return &v1.ConfigMap{}
    }
    isImmutable := func(object runtime.Object) bool {
        if configMap, ok := object.(*v1.ConfigMap); ok {
            return configMap.Immutable != nil && *configMap.Immutable
        }
        return false
    }
    gr := corev1.Resource("configmap")
    return &configMapManager{
        manager: manager.NewWatchBasedManager(listConfigMap, watchConfigMap, newConfigMap, isImmutable, gr, resyncInterval, getConfigMapNames),
    }
}

Secret Manager

secret manager 除了资源类型和 configmap 不一样，其他逻辑相同，所以仅列出两种 secret manager 的初始化函数。
/pkg /kubelet /secret /secret_manager.go

// NewCachingSecretManager creates a manager that keeps a cache of all secrets
// necessary for registered pods.
// It implements the following logic:
// - whenever a pod is created or updated, the cached versions of all secrets
//   are invalidated
// - every GetObject() call tries to fetch the value from local cache; if it is
//   not there, invalidated or too old, we fetch it from apiserver and refresh the
//   value in cache; otherwise it is just fetched from cache
func NewCachingSecretManager(kubeClient clientset.Interface, getTTL manager.GetObjectTTLFunc) Manager {
    getSecret := func(namespace, name string, opts metav1.GetOptions) (runtime.Object, error) {
        return kubeClient.CoreV1().Secrets(namespace).Get(context.TODO(), name, opts)
    }
    secretStore := manager.NewObjectStore(getSecret, clock.RealClock{}, getTTL, defaultTTL)
    return &secretManager{
        manager: manager.NewCacheBasedManager(secretStore, getSecretNames),
    }
}

// NewWatchingSecretManager creates a manager that keeps a cache of all secrets
// necessary for registered pods.
// It implements the following logic:
// - whenever a pod is created or updated, we start individual watches for all
//   referenced objects that aren't referenced from other registered pods
// - every GetObject() returns a value from local cache propagated via watches
func NewWatchingSecretManager(kubeClient clientset.Interface, resyncInterval time.Duration) Manager {
    listSecret := func(namespace string, opts metav1.ListOptions) (runtime.Object, error) {
        return kubeClient.CoreV1().Secrets(namespace).List(context.TODO(), opts)
    }
    watchSecret := func(namespace string, opts metav1.ListOptions) (watch.Interface, error) {
        return kubeClient.CoreV1().Secrets(namespace).Watch(context.TODO(), opts)
    }
    newSecret := func() runtime.Object {
        return &v1.Secret{}
    }
    isImmutable := func(object runtime.Object) bool {
        if secret, ok := object.(*v1.Secret); ok {
            return secret.Immutable != nil && *secret.Immutable
        }
        return false
    }
    gr := corev1.Resource("secret")
    return &secretManager{
        manager: manager.NewWatchBasedManager(listSecret, watchSecret, newSecret, isImmutable, gr, resyncInterval, getSecretNames),
    }
}

cache_based_manager

/pkg /kubelet /util /manager /cache_based_manager.go

// cacheBasedManager keeps a store with objects necessary
// for registered pods. Different implementations of the store
// may result in different semantics for freshness of objects
// (e.g. ttl-based implementation vs watch-based implementation).
type cacheBasedManager struct {
    objectStore          Store
    getReferencedObjects func(*v1.Pod) sets.String

    lock           sync.Mutex
    registeredPods map[objectKey]*v1.Pod
}

该 manager 代码位于 /pkg /kubelet /util /manager /cache_based_manager.go，属于通用的 Manager 结构体工具，用于保留注册的 Pod 所必要引用的 kubernetes 对象（objects）
如何做到的呢？
通过 getReferencedObjects 字段，一个可以传入的成员函数，自定义实现用于从 v1.Pod 对象中获取到对应对象（或一组对象）的 name。流程如下：

func (c *cacheBasedManager) RegisterPod(pod *v1.Pod) {
    // 1. 获取名字
    names := c.getReferencedObjects(pod)
    c.lock.Lock()
    defer c.lock.Unlock()
    // 2. 给每一个名字和 pod 的命名空间一起添加到 c.objectStore 中存储
    for name := range names {
        c.objectStore.AddReference(pod.Namespace, name)
    }
    // 3. 检查是否之前已经注册了该 Pod
    var prev *v1.Pod
    key := objectKey{namespace: pod.Namespace, name: pod.Name, uid: pod.UID}
    prev = c.registeredPods[key]
    // 4. 用新注册的 pod 替换之前存储的注册 Pod 的信息.
    c.registeredPods[key] = pod
    // 5. 删除旧 Pod 在 c.objectStore 中的引用信息,这是因为在上面第二步 Add 到 c.objectStore
    // 中时,这些资源的引用次数又新增了一次,但实际上只是同一个 Pod 的引用,自然需要删除,当然,也有
    // 可能新 Pod 已经不再引用目标资源了, Delete 函数在下面也处理这个情况
    if prev != nil {
        for name := range c.getReferencedObjects(prev) {
            // On an update, the .Add() call above will have re-incremented the
            // ref count of any existing object, so any objects that are in both
            // names and prev need to have their ref counts decremented. Any that
            // are only in prev need to be completely removed. This unconditional
            // call takes care of both cases.
            c.objectStore.DeleteReference(prev.Namespace, name)
        }
    }
}

1. 获取名字
2. 给每一个名字和 pod 的命名空间一起添加到 c.objectStore 中存储
3. 检查是否之前已经注册了该 Pod
4. 用新注册的 pod 替换之前存储的注册 Pod 的信息.
5. 删除旧 Pod 在 c.objectStore 中的引用信息,这是因为在上面第二步 Add 到 c.objectStore 中时,这些资源的引用次数又新增了一次,但实际上只是同一个 Pod 的引用,自然需要删除,当然,也有可能新 Pod 已经不再引用目标资源了, Delete 函数在下面也处理这个情况

ttl ObjectStore

cache_based 的 objectStore 通过 ttl 设置缓存有效期。

watch_based_manager

/pkg /kubelet /util /manager /watch_based_manager.go
可以看到，watch_based_manager 最后使用了 NewCacheBasedManager ，所以 watch_based_manager 和 cache_based_manager 不同的是 ObjectStore 字段。

// NewWatchBasedManager creates a manager that keeps a cache of all objects
// necessary for registered pods.
// It implements the following logic:
// - whenever a pod is created or updated, we start individual watches for all
//   referenced objects that aren't referenced from other registered pods
// - every GetObject() returns a value from local cache propagated via watches
func NewWatchBasedManager(
    listObject listObjectFunc,
    watchObject watchObjectFunc,
    newObject newObjectFunc,
    isImmutable isImmutableFunc,
    groupResource schema.GroupResource,
    resyncInterval time.Duration,
    getReferencedObjects func(*v1.Pod) sets.String) Manager {

    // If a configmap/secret is used as a volume, the volumeManager will visit the objectCacheItem every resyncInterval cycle,
    // We just want to stop the objectCacheItem referenced by environment variables,
    // So, maxIdleTime is set to an integer multiple of resyncInterval,
    // We currently set it to 5 times.
    maxIdleTime := resyncInterval * 5

    // TODO propagate stopCh from the higher level.
    objectStore := NewObjectCache(listObject, watchObject, newObject, isImmutable, groupResource, clock.RealClock{}, maxIdleTime, wait.NeverStop)
    return NewCacheBasedManager(objectStore, getReferencedObjects)
}

watch_based_manager 通过 watch 而不是简单的 ttl 去确认或者刷新缓存。